ml-supervised-learning

1 Dataset used:

Lending Club loan data - loans.csv Traffic sign image data - knn_traffic_signs.csv Donatiion data - donors.csv Brett’s location data - locations.csv

2 Four Classification Algorithms

This beginner-level introduction to machine learning covers four of the most common classification algorithms. You will come away with a basic understanding of how each algorithm approaches a learning task, as well as learn the R functions needed to apply these tools to your own work.

3 1: k-Nearest Neighbors (kNN)

As the kNN algorithm literally ‘learns by example’ it is a case in point for starting to understand supervised machine learning. This chapter will introduce classification while working through the application of kNN to self-driving vehicle road sign recognition.

• Classification with Nearest Neighbors
• Recognizing a road sign with kNN
• Thinking like kNN
• Exploring the traffic sign dataset
• Classifying a collection of road signs
• What about the ‘k’ in kNN
• Understanding the impact of ‘k’
• Testing other ‘k’ values
• Seeing how the neighbours voted
• Data preparation for kNN
• Why normalize data?

4 2: Naive Bayes

Naive Bayes uses principles form the field of statistics to make preditions. This chapter will introduce the basics of Bayesian methods while exploring how to apply these techniques to iPhone-like destination suggestions.

• Understanding Bayesian methods
• Computing probabilities
• Understanding dependent events
• A simple Naive Bayes location model
• Examining ‘raw’ probabilities
• Understanding independence
• Understanding NB’s naivety
• Who are you calling naive?
• A more sophisticated location model
• Preparing for unforseen circumstances
• Understanding the Laplace correction
• Applying Naive Bayes to other problems
• Handling numeric predictors

5 3: Logistics Regression

Logistic regression involves fitting a curve to numeric data to make predictions about binary events. Arguably one of the most widely used machine learning methods, this chapter will provide an overview of the technique while illustrating how to apply it to fundraising data.

• Making binary predictions with regression
• Building simple logistic regression models
• Making a binary prediction
• The limitations of accuracy
• Model performance tradeoffs
• Calculating ROC curves and AUC
• Comparing ROC curves
• Dummy variables, missing data, and interactions
• Coding categorical features
• Handling missing data
• Understanding missing value indicators
• Building a more sophisticated model
• Automatic feature selection
• The dangers of stepwise regression model
• Buidlign stepwise regression model

6 4: Classification Trees

Classification trees use flowchart-like structures to make decisions. Because humans can readily understand these tree structures, classification trees are useful when transparency is needed, such as in loan approval. We’ll use the Lending Club dataset to simulate this scenario.

• Making decision with trees
• Building a simple decsion tree
• Visualizing classification trees
• Understanding the tree’s decisions
• Growing larger classification trees
• Why do some branches split?
• Creating random test datasets
• Building and evaluating a larger tree
• Conducting a fair performance evaluation
• Tending to classification trees
• Preventing overgrown trees
• Creating a nicely pruned trees
• Why do trees benefit from pruning?
• Seeing the forest from trees
• Understanding random forests
• Building a random forest model

Instructor Brett Lanz - author book Machine Learning with R

7 Classification with Nearest Neighbors

• ML utilizes computers to turn data into insight and action
• this course focuses on a subset of ML
• the sub-domain called supervised learning
• focuses on training the machine to learn from prior examples
• when the concept to be learned is a set of categories
• the task is called classification
• from identifying diseases to predicting the weather or detecting an image contains a cat
• classification tasks are diverse yet common
• in this course we will learn classification methods while exploring four real world applications
• let’s get started

8 Classification tasks for driverless cars

• if your experiences on the road are anything like mine
• self driving cars can’t get here soon enough
• it is easy to imagine aspects of autonomous driving that involve classification
• for example, when a vehicles camera observes an object it must classify the object before it can react
• although the algorithms that govern the autonomous cars are sophisticated we can simulate the aspects for their behaviour
• in this example we will suppose that the vehicle can see but not distinguish the roadway signs
• your job will be to use ML to classify the signs type
• to start training a self driving car you might supervise it by demonstrating the desired behaviour as it observes each type of sign
• you stop at intersections, yield to pedestrians, and change speed as needed
• after some time on your instruction, the vehicle has built a database that records the sign as well as the target behaviour

images of stops signs(mostly red), walk sign (mostly green), adjust speed signs (mostly bluish grey and number in black) * the image here illustrates the dataset * I suspect you already see some similarities, the machine can too * a nearest neighbour classifier takes advantage of the fact that the signs that look alike should be similar to or nearby other signs of the same type * for example if the car observes a sign that seems similar to those in the group of stop sign the will probably need to stop * so how does nearest neighbour learner decide whether two signs are similar * it does so by literally measuring the distance between them * that not to say that it measures distance between the signs in the physical space * a stop sign in New York is the same as the stop sign in Los Angeles * but instead it imagines the properties of the signs in coordinates what is called a feature space * consider for instance a signs colour * by imagining the colour as a three dimensional feature space measuring levels of red, green and blue * signs of similar colour are located naturally close to one another * once the feature space has been constructed in this way you can measure distance using formulat like

dist(p,q) = square root of sum of squares of p1-q1, p2-q2, …pn-qn

• many nearest neighbour learners use euclidean distance formula here
• which measures the straight line distance between two points
• we don’t have to worry about the formula as R will compute it for us
• an algorithm called K nearest neighbour or KNN
• uses the principle of nearest neighbours to classify unlabeled examples
• we’ll get into the specifics later but for now it suffices to know that by default R’s knn fucntion searches the dataset for the observations most similar to the new observed one
• the knn function is the part of the class package
• and requires three parameters
• frist the set of training data, second the test data to be classified and third the labels for the training data

9 Recognizing a road sign with kNN

After several trips with a human behind the wheel, it is time for the self-driving car to attempt the test course alone.

As it begins to drive away, its camera captures the following image:

Stop Sign

Can you apply a kNN classifier to help the car recognize this sign?

install.packages("dplyr")
install.packages("readr")
install.packages("ggplot2")
install.packages("purrr")
install.packages("class")

library(dplyr)

## Warning: package 'dplyr' was built under R version 3.5.3

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(readr)

## Warning: package 'readr' was built under R version 3.5.3

library(ggplot2)

## Warning: package 'ggplot2' was built under R version 3.5.3

library(purrr)

## Warning: package 'purrr' was built under R version 3.5.3

library(class)

## Warning: package 'class' was built under R version 3.5.3

signs = read_csv("C:/shobha/R/DataCamp/dataFiles/CSV-files/knn_traffic_signs.csv")

## Parsed with column specification:
## cols(
##   .default = col_double(),
##   sample = col_character(),
##   sign_type = col_character()
## )

## See spec(...) for full column specifications.

glimpse(signs)

## Observations: 206
## Variables: 51
## $ id        <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1...
## $ sample    <chr> "train", "train", "train", "train", "train", "train"...
## $ sign_type <chr> "pedestrian", "pedestrian", "pedestrian", "pedestria...
## $ r1        <dbl> 155, 142, 57, 22, 169, 75, 136, 118, 149, 13, 123, 1...
## $ g1        <dbl> 228, 217, 54, 35, 179, 67, 149, 105, 225, 34, 124, 1...
## $ b1        <dbl> 251, 242, 50, 41, 170, 60, 157, 69, 241, 28, 107, 13...
## $ r2        <dbl> 135, 166, 187, 171, 231, 131, 200, 244, 34, 5, 83, 3...
## $ g2        <dbl> 188, 204, 201, 178, 254, 89, 203, 245, 45, 21, 61, 4...
## $ b2        <dbl> 101, 44, 68, 26, 27, 53, 107, 67, 1, 11, 26, 37, 26,...
## $ r3        <dbl> 156, 142, 51, 19, 97, 214, 150, 132, 155, 123, 116, ...
## $ g3        <dbl> 227, 217, 51, 27, 107, 144, 167, 123, 226, 154, 124,...
## $ b3        <dbl> 245, 242, 45, 29, 99, 75, 134, 12, 238, 140, 115, 12...
## $ r4        <dbl> 145, 147, 59, 19, 123, 156, 171, 138, 147, 21, 67, 4...
## $ g4        <dbl> 211, 219, 62, 27, 147, 169, 218, 123, 222, 46, 67, 5...
## $ b4        <dbl> 228, 242, 65, 29, 152, 190, 252, 85, 242, 41, 52, 49...
## $ r5        <dbl> 166, 164, 156, 42, 221, 67, 171, 254, 170, 36, 70, 1...
## $ g5        <dbl> 233, 228, 171, 37, 236, 50, 158, 254, 191, 60, 53, 1...
## $ b5        <dbl> 245, 229, 50, 3, 117, 36, 108, 92, 113, 26, 26, 141,...
## $ r6        <dbl> 212, 84, 254, 217, 205, 37, 157, 241, 26, 75, 26, 60...
## $ g6        <dbl> 254, 116, 255, 228, 225, 36, 186, 240, 37, 108, 26, ...
## $ b6        <dbl> 52, 17, 36, 19, 80, 42, 11, 108, 12, 44, 21, 18, 20,...
## $ r7        <dbl> 212, 217, 211, 221, 235, 44, 26, 254, 34, 13, 52, 9,...
## $ g7        <dbl> 254, 254, 226, 235, 254, 42, 35, 254, 45, 27, 45, 13...
## $ b7        <dbl> 11, 26, 70, 20, 60, 44, 10, 99, 19, 25, 27, 17, 20, ...
## $ r8        <dbl> 188, 155, 78, 181, 90, 192, 180, 108, 221, 133, 117,...
## $ g8        <dbl> 229, 203, 73, 183, 110, 131, 211, 106, 249, 163, 109...
## $ b8        <dbl> 117, 128, 64, 73, 9, 73, 236, 27, 184, 126, 83, 33, ...
## $ r9        <dbl> 170, 213, 220, 237, 216, 123, 129, 135, 226, 83, 110...
## $ g9        <dbl> 216, 253, 234, 234, 236, 74, 109, 123, 246, 125, 74,...
## $ b9        <dbl> 120, 51, 59, 44, 66, 22, 73, 40, 59, 19, 12, 12, 18,...
## $ r10       <dbl> 211, 217, 254, 251, 229, 36, 161, 254, 30, 13, 98, 2...
## $ g10       <dbl> 254, 255, 255, 254, 255, 34, 190, 254, 40, 27, 70, 1...
## $ b10       <dbl> 3, 21, 51, 2, 12, 37, 10, 115, 34, 25, 26, 11, 20, 2...
## $ r11       <dbl> 212, 217, 253, 235, 235, 44, 161, 254, 34, 9, 20, 28...
## $ g11       <dbl> 254, 255, 255, 243, 254, 42, 190, 254, 44, 23, 21, 2...
## $ b11       <dbl> 19, 21, 44, 12, 60, 44, 6, 99, 35, 18, 20, 19, 13, 1...
## $ r12       <dbl> 172, 158, 66, 19, 163, 197, 187, 138, 241, 85, 113, ...
## $ g12       <dbl> 235, 225, 68, 27, 168, 114, 215, 123, 255, 128, 76, ...
## $ b12       <dbl> 244, 237, 68, 29, 152, 21, 236, 85, 54, 21, 14, 12, ...
## $ r13       <dbl> 172, 164, 69, 20, 124, 171, 141, 118, 205, 83, 106, ...
## $ g13       <dbl> 235, 227, 65, 29, 117, 102, 142, 105, 229, 125, 69, ...
## $ b13       <dbl> 244, 237, 59, 34, 91, 26, 140, 75, 46, 19, 9, 12, 13...
## $ r14       <dbl> 172, 182, 76, 64, 188, 197, 189, 131, 226, 85, 102, ...
## $ g14       <dbl> 228, 228, 84, 61, 205, 114, 171, 124, 246, 128, 67, ...
## $ b14       <dbl> 235, 143, 22, 4, 78, 21, 140, 5, 59, 21, 6, 12, 13, ...
## $ r15       <dbl> 177, 171, 82, 211, 125, 123, 214, 106, 235, 85, 106,...
## $ g15       <dbl> 235, 228, 93, 222, 147, 74, 221, 94, 252, 128, 69, 4...
## $ b15       <dbl> 244, 196, 17, 78, 20, 22, 201, 53, 67, 21, 9, 11, 18...
## $ r16       <dbl> 22, 164, 58, 19, 160, 180, 188, 101, 237, 83, 43, 60...
## $ g16       <dbl> 52, 227, 60, 27, 183, 107, 211, 91, 254, 125, 29, 45...
## $ b16       <dbl> 53, 237, 60, 29, 187, 26, 227, 59, 53, 19, 11, 18, 1...

Remove the id and sample columns from signs dataset

#signs = signs %>% select(-id, -sample) 
#or
#signs = signs[, -c(1,2)]
#or
signs = signs[, -(1:2)]
names(signs)

##  [1] "sign_type" "r1"        "g1"        "b1"        "r2"       
##  [6] "g2"        "b2"        "r3"        "g3"        "b3"       
## [11] "r4"        "g4"        "b4"        "r5"        "g5"       
## [16] "b5"        "r6"        "g6"        "b6"        "r7"       
## [21] "g7"        "b7"        "r8"        "g8"        "b8"       
## [26] "r9"        "g9"        "b9"        "r10"       "g10"      
## [31] "b10"       "r11"       "g11"       "b11"       "r12"      
## [36] "g12"       "b12"       "r13"       "g13"       "b13"      
## [41] "r14"       "g14"       "b14"       "r15"       "g15"      
## [46] "b15"       "r16"       "g16"       "b16"

# creating test observatiion to predict label using knn
next_sign = signs[206, -1]

glimpse(next_sign)

## Observations: 1
## Variables: 48
## $ r1  <dbl> 204
## $ g1  <dbl> 227
## $ b1  <dbl> 220
## $ r2  <dbl> 196
## $ g2  <dbl> 59
## $ b2  <dbl> 51
## $ r3  <dbl> 202
## $ g3  <dbl> 67
## $ b3  <dbl> 59
## $ r4  <dbl> 204
## $ g4  <dbl> 227
## $ b4  <dbl> 220
## $ r5  <dbl> 236
## $ g5  <dbl> 250
## $ b5  <dbl> 234
## $ r6  <dbl> 242
## $ g6  <dbl> 252
## $ b6  <dbl> 235
## $ r7  <dbl> 205
## $ g7  <dbl> 148
## $ b7  <dbl> 131
## $ r8  <dbl> 190
## $ g8  <dbl> 50
## $ b8  <dbl> 43
## $ r9  <dbl> 179
## $ g9  <dbl> 70
## $ b9  <dbl> 57
## $ r10 <dbl> 242
## $ g10 <dbl> 229
## $ b10 <dbl> 212
## $ r11 <dbl> 190
## $ g11 <dbl> 50
## $ b11 <dbl> 43
## $ r12 <dbl> 193
## $ g12 <dbl> 51
## $ b12 <dbl> 44
## $ r13 <dbl> 170
## $ g13 <dbl> 197
## $ b13 <dbl> 196
## $ r14 <dbl> 190
## $ g14 <dbl> 50
## $ b14 <dbl> 43
## $ r15 <dbl> 190
## $ g15 <dbl> 47
## $ b15 <dbl> 41
## $ r16 <dbl> 165
## $ g16 <dbl> 195
## $ b16 <dbl> 196

Create a vector of sign labels to use with kNN by extracting the column sign_type from signs.

Identify the next_sign using the knn() function.

Set the train argument equal to the signs data frame without the first column.

Set the test argument equal to the data frame next_sign.

Use the vector of labels you created as the cl argument.

# Load the 'class' package

# Create a vector of labels
sign_types <- signs$sign_type

str(sign_types)

##  chr [1:206] "pedestrian" "pedestrian" "pedestrian" "pedestrian" ...

# chr [1:206] "pedestrian" "pedestrian" "pedestrian" "pedestrian" "pedestrian" ...

# Classify the next sign observed
knn(train = signs[-1], test = next_sign, cl = sign_types)

## [1] stop
## Levels: pedestrian speed stop

Awesome! You’ve trained your first nearest neighbor classifier!

Thinking like kNN With your help, the test car successfully identified the sign and stopped safely at the intersection.

How did the knn() function correctly classify the stop sign?

Possible Answers It learned that stop signs are red

The sign was in some way similar to another stop sign (answer)

Stop signs have eight sides

The other types of signs were less likely

Correct! kNN isn’t really learning anything; it simply looks for the most similar example.

10 Exploring the traffic sign dataset

To better understand how the knn() function was able to classify the stop sign, it may help to examine the training dataset it used.

Each previously observed street sign was divided into a 4x4 grid, and the red, green, and blue level for each of the 16 center pixels is recorded as illustrated here.

The result is a dataset that records the sign_type as well as 16 x 3 = 48 color properties of each sign.

Knn Stop Sign

blue part - red:204 green:227 blue:220 red part - red:193 green :52 blue:44

Use the str() function to examine the signs dataset.

Use table() to count the number of observations of each sign type by passing it the column containing the labels.

Run the provided aggregate() command to see whether the average red level might vary by sign type.

# Examine the structure of the signs dataset
str(signs)

## Classes 'tbl_df', 'tbl' and 'data.frame':    206 obs. of  49 variables:
##  $ sign_type: chr  "pedestrian" "pedestrian" "pedestrian" "pedestrian" ...
##  $ r1       : num  155 142 57 22 169 75 136 118 149 13 ...
##  $ g1       : num  228 217 54 35 179 67 149 105 225 34 ...
##  $ b1       : num  251 242 50 41 170 60 157 69 241 28 ...
##  $ r2       : num  135 166 187 171 231 131 200 244 34 5 ...
##  $ g2       : num  188 204 201 178 254 89 203 245 45 21 ...
##  $ b2       : num  101 44 68 26 27 53 107 67 1 11 ...
##  $ r3       : num  156 142 51 19 97 214 150 132 155 123 ...
##  $ g3       : num  227 217 51 27 107 144 167 123 226 154 ...
##  $ b3       : num  245 242 45 29 99 75 134 12 238 140 ...
##  $ r4       : num  145 147 59 19 123 156 171 138 147 21 ...
##  $ g4       : num  211 219 62 27 147 169 218 123 222 46 ...
##  $ b4       : num  228 242 65 29 152 190 252 85 242 41 ...
##  $ r5       : num  166 164 156 42 221 67 171 254 170 36 ...
##  $ g5       : num  233 228 171 37 236 50 158 254 191 60 ...
##  $ b5       : num  245 229 50 3 117 36 108 92 113 26 ...
##  $ r6       : num  212 84 254 217 205 37 157 241 26 75 ...
##  $ g6       : num  254 116 255 228 225 36 186 240 37 108 ...
##  $ b6       : num  52 17 36 19 80 42 11 108 12 44 ...
##  $ r7       : num  212 217 211 221 235 44 26 254 34 13 ...
##  $ g7       : num  254 254 226 235 254 42 35 254 45 27 ...
##  $ b7       : num  11 26 70 20 60 44 10 99 19 25 ...
##  $ r8       : num  188 155 78 181 90 192 180 108 221 133 ...
##  $ g8       : num  229 203 73 183 110 131 211 106 249 163 ...
##  $ b8       : num  117 128 64 73 9 73 236 27 184 126 ...
##  $ r9       : num  170 213 220 237 216 123 129 135 226 83 ...
##  $ g9       : num  216 253 234 234 236 74 109 123 246 125 ...
##  $ b9       : num  120 51 59 44 66 22 73 40 59 19 ...
##  $ r10      : num  211 217 254 251 229 36 161 254 30 13 ...
##  $ g10      : num  254 255 255 254 255 34 190 254 40 27 ...
##  $ b10      : num  3 21 51 2 12 37 10 115 34 25 ...
##  $ r11      : num  212 217 253 235 235 44 161 254 34 9 ...
##  $ g11      : num  254 255 255 243 254 42 190 254 44 23 ...
##  $ b11      : num  19 21 44 12 60 44 6 99 35 18 ...
##  $ r12      : num  172 158 66 19 163 197 187 138 241 85 ...
##  $ g12      : num  235 225 68 27 168 114 215 123 255 128 ...
##  $ b12      : num  244 237 68 29 152 21 236 85 54 21 ...
##  $ r13      : num  172 164 69 20 124 171 141 118 205 83 ...
##  $ g13      : num  235 227 65 29 117 102 142 105 229 125 ...
##  $ b13      : num  244 237 59 34 91 26 140 75 46 19 ...
##  $ r14      : num  172 182 76 64 188 197 189 131 226 85 ...
##  $ g14      : num  228 228 84 61 205 114 171 124 246 128 ...
##  $ b14      : num  235 143 22 4 78 21 140 5 59 21 ...
##  $ r15      : num  177 171 82 211 125 123 214 106 235 85 ...
##  $ g15      : num  235 228 93 222 147 74 221 94 252 128 ...
##  $ b15      : num  244 196 17 78 20 22 201 53 67 21 ...
##  $ r16      : num  22 164 58 19 160 180 188 101 237 83 ...
##  $ g16      : num  52 227 60 27 183 107 211 91 254 125 ...
##  $ b16      : num  53 237 60 29 187 26 227 59 53 19 ...

# Count the number of signs of each type
table(signs$sign_type)

## 
## pedestrian      speed       stop 
##         65         70         71

# Check r10's average red level by sign type 
# average of red colour in each of the sign types
aggregate(r10 ~ sign_type, data = signs, mean)

##    sign_type       r10
## 1 pedestrian 108.78462
## 2      speed  83.08571
## 3       stop 142.50704

# same can be done achieved as below using dplyr verbs
signs %>% 
  group_by(sign_type) %>%
  summarise(mean(r10))

## # A tibble: 3 x 2
##   sign_type  `mean(r10)`
##   <chr>            <dbl>
## 1 pedestrian       109. 
## 2 speed             83.1
## 3 stop             143.

Great work! As you might have expected, stop signs tend to have a higher average red value. This is how kNN identifies similar signs.

11 Classifying a collection of road signs

Now that the autonomous vehicle has successfully stopped on its own, your team feels confident allowing the car to continue the test course.

The test course includes 59 additional road signs divided into three types:

Speed Sign

Ped Sign

Stop Sign

At the conclusion of the trial, you are asked to measure the car’s overall performance at recognizing these signs.

The class package and the dataset signs are already loaded in your workspace. So is the dataframe test_signs, which holds a set of observations you’ll test your model on.

12 Classify the test_signs data using knn():

Set train equal to the observations in signs without labels.

Use test_signs for the test argument, again without labels.

For the cl argument, use the vector of labels provided for you.

Use table() to explore the classifier’s performance at identifying the three sign types.

Create the vector signs_actual by extracting the labels from test_signs.

Pass the vector of predictions and the vector of actual signs to table() to cross tabulate them.

Compute the overall accuracy of the kNN learner using the mean() function.

test_index = c(8,13,14,19,20,22,29,30,36,44,45,46,
                       47,50,52,53,57,62,63,66,67,69,74,75,
                       78,82,84,100,101,103,110,113,117,123,124,130,
                       131,132,133,135,137,140,142,143,148,151,154,156,
                       157,164,174,175,181,183,192,193,201,203,205)

test_signs = signs[test_index,]
dim(test_signs)

## [1] 59 49

#[1] 59 49

train_signs = signs[-test_index,]
dim(train_signs)

## [1] 147  49

#[1] 147  49

dim(signs)

## [1] 206  49

#[1] 206  49

# Use kNN to identify the test road signs
sign_types = train_signs$sign_type
signs_pred = knn(train = train_signs[, -1], 
                 test = test_signs[-1], 
                 cl = sign_types)


# Create a confusion matrix of the actual versus predicted values
signs_actual = test_signs$sign_type
table(signs_actual,signs_pred)

##             signs_pred
## signs_actual pedestrian speed stop
##   pedestrian         19     0    0
##   speed               2    17    2
##   stop                0     0   19

# Compute the accuracy
mean(signs_actual ==  signs_pred)

## [1] 0.9322034

Fantastic! That self-driving car is really coming along! The confusion matrix lets you look for patterns in the classifier’s errors.

13 What about the ‘k’ in kNN

• you must be wondering why kNN is called k nearest neighbours
• what exactly is k
• the letter ‘k’ specifies the number of neighbours to consider when making the classification
• you can think of it as determining the size of the neighbourhoods
• until now we have ignored k thus R has used the defaulted value of 1
• this means the single nearest most similar neighbour was used to classify the unlabeled example
• while this looks okay on the surface let’s work through an example to see why the value of k may have a substantial impact on the performance of our classifier

KNN Neighbors

• suppose our vehicle observed a pedestrian sign, as shown in the image, and its five nearest neighbours are depicted
• the single nearest neighbour is the speed limit sign which shares a very similar background colour
• unfortunately in this case a knn classifier with k set to 1 would make an incorrect classification
• sightly further away are the 2nd, 3rd and 4th nearest neighbours
• which are all pedestrian crossing signs
• suppose we set k to 3, what would happen
• the three nearest neighbours, a speed limit sign and two pedestrian crossing signs would take a vote
• the category with the majority of nearest neighbours, in this case, the pedestrian crossing sign, is the winner
• increasing k to 5 allows the five nearest neighbours to vote
• the pedestrian crossing sign still wins by a margin of three to two
• note in the case of a tie the winner is typically decided at random

Bigger ‘k’ is not always better

• in the previous example setting k to a bigger value resulted in a correct prediction
• but it is not always the case the case that bigger is better
• a small k creates a vary small neighbourhoods, the classifier is able to discover very subtle patterns

KNN Impact Small

• as this image illustrates, it is able to distinguish groups even when their boundary is somewhat fuzzy
• on the other hand sometimes a fuzzy bounday is not a true pattern but due to some other factor that adds randomness to the data this is called noise

KNN Impact Large

• setting k larger, as this image shows, ignores some potentially noisy points in an effort to discover a broader more general pattern
• so how should you set k
• unfortunately, there is no universal rule
• in practise the optimal value depends on the complexity of the pattern to be learned
• as well as the impact of the noisy data, some suggest a rule of thumb starting with k equal to the square root of the number of observations in the training data
• for example, if the car had observed 100 previous road signs you might set k to 10
• an even better approach is test several different values of k and compare the performance on the data it has not seen before
• we will see the impact of k on the vehicles ability to correctly classify signs

14 Understanding the impact of ‘k’

There is a complex relationship between k and classification accuracy. Bigger is not always better.

Which of these is a valid reason for keeping k as small as possible (but no smaller)?

ANSWER THE QUESTION

Possible Answers A smaller k requires less processing power

A smaller k reduces the impact of noisy data

A smaller k minimizes the chance of a tie vote

A smaller k may utilize more subtle patterns (answer)

Yes! With smaller neighborhoods, kNN can identify more subtle patterns in the data.

Testing other ‘k’ values By default, the knn() function in the class package uses only the single nearest neighbor.

Setting a k parameter allows the algorithm to consider additional nearby neighbors. This enlarges the collection of neighbors which will vote on the predicted class.

Compare k values of 1, 7, and 15 to examine the impact on traffic sign classification accuracy.

#cl implies vector of lables
# sign prediction with k = 1
k_1 = knn(train = train_signs[-1], 
          test = test_signs[-1], 
          cl = train_signs$sign_type,
          k = 1)

k_1

##  [1] pedestrian pedestrian pedestrian pedestrian pedestrian pedestrian
##  [7] pedestrian pedestrian pedestrian pedestrian pedestrian pedestrian
## [13] pedestrian pedestrian pedestrian pedestrian pedestrian pedestrian
## [19] pedestrian stop       pedestrian speed      speed      speed     
## [25] speed      speed      speed      stop       pedestrian speed     
## [31] speed      speed      speed      speed      speed      speed     
## [37] speed      speed      speed      speed      stop       stop      
## [43] stop       stop       stop       stop       stop       stop      
## [49] stop       stop       stop       stop       stop       stop      
## [55] stop       stop       stop       stop       stop      
## Levels: pedestrian speed stop

# accuracy
mean(signs_actual == k_1)

## [1] 0.9322034

# sign prediction with k = 7
k_7 = knn(train = train_signs[-1], 
          test = test_signs[-1], 
          cl = train_signs$sign_type,
          k = 7)

k_7

##  [1] pedestrian pedestrian pedestrian stop       pedestrian pedestrian
##  [7] pedestrian pedestrian pedestrian pedestrian pedestrian pedestrian
## [13] pedestrian pedestrian pedestrian pedestrian pedestrian pedestrian
## [19] pedestrian speed      speed      speed      speed      speed     
## [25] speed      speed      speed      stop       speed      speed     
## [31] speed      speed      speed      speed      speed      speed     
## [37] speed      speed      speed      speed      stop       stop      
## [43] stop       stop       stop       stop       stop       stop      
## [49] stop       stop       stop       stop       stop       stop      
## [55] stop       stop       stop       stop       stop      
## Levels: pedestrian speed stop

# accuracy
mean(signs_actual == k_7)

## [1] 0.9661017

#sign prediction with k = 15
k_15 = knn(train = train_signs[-1], 
          test = test_signs[-1], 
          cl = train_signs$sign_type,
          k = 15)

k_15

##  [1] pedestrian stop       pedestrian stop       stop       pedestrian
##  [7] pedestrian pedestrian pedestrian pedestrian pedestrian pedestrian
## [13] pedestrian speed      pedestrian pedestrian pedestrian pedestrian
## [19] stop       speed      speed      speed      speed      speed     
## [25] speed      speed      speed      stop       speed      speed     
## [31] speed      speed      speed      speed      speed      speed     
## [37] speed      speed      speed      speed      stop       stop      
## [43] stop       stop       stop       stop       stop       stop      
## [49] stop       stop       stop       stop       stop       stop      
## [55] stop       stop       stop       stop       stop      
## Levels: pedestrian speed stop

# accuracy
mean(signs_actual == k_15)

## [1] 0.8983051

You’re a kNN pro! Which value of k gave the highest accuracy? k = 7

Seeing how the neighbors voted When multiple nearest neighbors hold a vote, it can sometimes be useful to examine whether the voters were unanimous or widely separated.

For example, knowing more about the voters’ confidence in the classification could allow an autonomous vehicle to use caution in the case there is any chance at all that a stop sign is ahead.

In this exercise, you will learn how to obtain the voting results from the knn() function.

Build a kNN model with the prob = TRUE parameter to compute the vote proportions. Set k = 7. Use the attr() function to obtain the vote proportions for the predicted class. These are stored in the attribute ‘prob’. Examine the first several vote outcomes and percentages using the head() function to see how the confidence varies from sign to sign.

# Use the prob parameter to get the proportion of votes for the winning class
sign_pred_k_7 = knn(train = train_signs[-1],
          test = test_signs[-1],
          cl = train_signs$sign_type,
          k = 7,
          prob = TRUE)



# Get the "prob" attribute from the predicted classes
sign_prob = attr(sign_pred_k_7, "prob")


# Examine the first several predictions
head(sign_pred_k_7)

## [1] pedestrian pedestrian pedestrian stop       pedestrian pedestrian
## Levels: pedestrian speed stop

"
[1] pedestrian pedestrian pedestrian stop       pedestrian pedestrian
Levels: pedestrian speed stop
"

## [1] "\n[1] pedestrian pedestrian pedestrian stop       pedestrian pedestrian\nLevels: pedestrian speed stop\n"

# Examine the proportion of votes for the winning class
head(sign_prob)

## [1] 0.5714286 0.5714286 0.8571429 0.5714286 0.8571429 0.5714286

#[1] 0.5714286 0.5714286 0.8571429 0.5714286 0.8571429 0.5714286

Wow! Awesome job! Now you can get an idea of how certain your kNN learner is about its classifications.

15 Data preparation for KNN

• you have now seen the knn algorithm in action while simulating aspects of a self driving vehicle
• you’ve gained an understanding of impact of k on the algorithms performance and know how to examine neighbour’s votes which predictions are closer to unanimous
• but before applying knn to your own projects, you’ll need to know one more thing
• how to prepare your data for nearest neighbours

16 kNN assumes numeric data

• as noted earlier nearest neighbour learners use distance functions to identify the most similar or nearest examples
• many common distance functions assume that your data are in numeric format as it is difficult to define the distance between categories
• for example there is no obvious way to define the distance between red and yellow
• consequently the traffic signs dataset represented these as numeric colour intensities
• but suppose that you had a property that cannot be measured numerically
• such as whether the road sign is rectangle diamond or octegon
• a common solution is to use 1 0 to represent these categories
• this is called dummy coding

speed limit sign - rectangle = 1, diamond = 0 pedestrian crossing - rectangle = 1, diamond = 1 stop sign - rectangle = 0, diamond = 0

• a binary dummy variable is created for each category except one
• this variable is set to 1 if category applies otherwise 0
• the category that is left out can be easily deduced, if stop sign is not a rectangle or a diamond then it must be an octegon
• dummy coded data can be used directly in a distance function
• two rectangle signs both having values of 1 will be found to be closer together than a rectangle and a diamond

17 kNN benefits from normalized data

• it is also important to be aware that while calculating distance
• each feature of the input data should be measured with the same range of values
• this was true for the traffic sign data
• each colour code ranged from a min of zero to a max of 255
• however, suppose that we added 1,0 dummy variables for signs shapes into the distance calculation
• two different shapes may differ from one another by atmost one unit but two colours may differ from as much as 255 units
• such a difference allows features with a wider range to have more influence over the distance calculation, see

KNN Normalize Before

• here the topmost speed limit sign is closer to the pedestrian sign than it is to its correct neighbours {because it is bluer than other two speed limit signs}
• simply because the range of blue values is wider than the 0 to 1 range of shape values
• compressing the blue axes, so that it also follows the 0 to 1 range corrects this issue and the speed limit sign is now closer to its neighbourhood

KNN Normalize After

18 Normalizing data in R

• R does not have a builtin fucntion to rescale data to a given range
• so you’ll need to create one yourself
• the below function performs the min max normalization
• this rescales the vector x such that its min value is 0 and max value is 1
• it does so by subracting the min value of x from each value of x and dividing by range of x
• after applying this function to r1, one of the colour vectors, we can use the summary function to see that the new min and max values are 0 and 1 respectively
• calcuating the summary statitics for the same un-normalized data shows a min of 3 and a max of 251

# define a min-max normalize() function
normalize = function(x){
    return((x - min(x)) /  (max(x) - min(x)))
}

# normalized version of r1
summary(normalize(signs$r1))

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##  0.0000  0.1935  0.3528  0.4046  0.6129  1.0000

# un-normalized version of r1
summary(signs$r1)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##     3.0    51.0    90.5   103.3   155.0   251.0

19 Why normalize data?

Before applying kNN to a classification task, it is common practice to rescale the data using a technique like min-max normalization. What is the purpose of this step?

ANSWER THE QUESTION

Possible Answers To ensure all data elements may contribute equal shares to distance. (answer)

To help the kNN algorithm converge on a solution faster.

To convert all of the data elements to numbers.

To redistribute the data as a normal bell curve.

Yes! Rescaling reduces the influence of extreme values on kNN’s distance function.

20 Understanding Bayesian methods

• some smart phones and apps predict the users destination to offer routes and traffic estimates without the user even asking them
• if you didn’t know about machine learning, the phones ability to predict the future this way might seem a bit like magic
• the phone obviously keeps a record of the users past locations, it then uses this data to forecast the users most probable future location much like a meteorologists estimates the precipitation probability in a weather report
• a branch of statistics called Bayesian methods applies the work of 18th century statistician Thomas Bayes who proposed rules for estimating probabilities in light of historic data
• by applying these methods to my own location tracking data you will learn how probability estimates can forecast action

21 Estimating probability

• let’s see where the data finds me

NN Locations

• this map show the number of times my phone recorded my position at four different locations
• based on this data my phone can predict that at any given time my most probable location is at work, because I was there 57.5% of the time

The probability of A is denoted P(A)

P(work) = 23 / 40 = 57.5% P(store) = 4 / 40 = 10.0%

• 23 of the past 40 times it checked
• this illustrates how probability of an event is estimated from historic data
• it is the number of times an event happened divided by the number of times it could have happened
• but eventhough I’m at work a lot, the phone should not predict I’m there all the time
• instead it should incorporate additional data like time of day to better tailor its prediction ot the situation
• this requires an understanding of how to combine information from several events into a single probability estimate

Joint probaility and independent events * when events occur together they have a joint probability their intersection can be depicted using a venn diagram * like those shown here

NB Venn

• these show there is much greater probability that I’m at work in the afternoon than in the evening
• the overlap is much greater for work in the afternoon
• the joint probability of two events is computed by finding the proportion of observations in which they occured together

The joint probability of events A and B is denoted P(A and B)

P(work and evening) = 1% P(work and afternoon) = 20%

• sometimes one event does not influence the probability of another, these are said to be independent events
• for example my location is unrelated to most other users locations
• knowing where they are does not provide information about where I might be
• this notion of independent events would be important later on, however lot of the other data elements that my phone collects such as time and date are very predictive of where I might be
• when one event is predictive of another they are called dependent and these are the basis of prediction in Bayesian methods

Conditional probability and dependent events

NB Venn

The conditional probability of events A and B is denoted P(A|B) - it is their joint probability divided by probability of B

P(A|B) = P(A and B)/P(B)

P(work | evening) = 1/25 = 4% P(work | afternoon) = 20/25 = 80%

• conditional probability expresses exactly how one event depends on another

22 Making predictions with Naive Bayes

• the algorithm known as Naive Bayes applies Bayesian methods to estimate the conditional probabilities of an outcome
• the naivebayes package provides the functions to build this model

# building a Naive Bayes model
library(naivebayes)

#m = naive_bayes(location ~ time_of_day, data = location_history)

# making predictions with Naive Bayes
#future_location = predict(m, future_conditions)

# Create data frame as below from locations.csv
#str(locations)

• since location depends on time of day
• the corresponding predict function computes conditional probabilities to predict a future location based on the future conditions

library(naivebayes)

locations_org = read_csv("C:/shobha/R/DataCamp/dataFiles/CSV-files/locations.csv")

## Parsed with column specification:
## cols(
##   month = col_double(),
##   day = col_double(),
##   weekday = col_character(),
##   daytype = col_character(),
##   hour = col_double(),
##   hourtype = col_character(),
##   location = col_character()
## )

dim(locations_org)

## [1] 2184    7

glimpse(locations_org)

## Observations: 2,184
## Variables: 7
## $ month    <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...
## $ day      <dbl> 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,...
## $ weekday  <chr> "wednesday", "wednesday", "wednesday", "wednesday", "...
## $ daytype  <chr> "weekday", "weekday", "weekday", "weekday", "weekday"...
## $ hour     <dbl> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,...
## $ hourtype <chr> "night", "night", "night", "night", "night", "night",...
## $ location <chr> "home", "home", "home", "home", "home", "home", "home...

length(unique(locations_org$day))

## [1] 31

length(unique(locations_org$month))

## [1] 4

length(unique(locations_org$weekday))

## [1] 7

length(unique(locations_org$daytype))

## [1] 2

length(unique(locations_org$hour))

## [1] 24

length(unique(locations_org$hourtype))

## [1] 4

length(unique(locations_org$location))

## [1] 7

unique(locations_org$month)

## [1] 1 2 3 4

unique(locations_org$hourtype)

## [1] "night"     "morning"   "afternoon" "evening"

unique(locations_org$hour)

##  [1]  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22
## [24] 23

head(locations_org)

## # A tibble: 6 x 7
##   month   day weekday   daytype  hour hourtype location
##   <dbl> <dbl> <chr>     <chr>   <dbl> <chr>    <chr>   
## 1     1     4 wednesday weekday     0 night    home    
## 2     1     4 wednesday weekday     1 night    home    
## 3     1     4 wednesday weekday     2 night    home    
## 4     1     4 wednesday weekday     3 night    home    
## 5     1     4 wednesday weekday     4 night    home    
## 6     1     4 wednesday weekday     5 night    home

length(unique(locations_org$month))

## [1] 4

length(unique(locations_org$weekday))

## [1] 7

length(unique(locations_org$daytype))

## [1] 2

length(unique(locations_org$hour))

## [1] 24

length(unique(locations_org$hourtype))

## [1] 4

length(unique(locations_org$location))

## [1] 7

length(unique(locations_org$day))

## [1] 31

unique(locations_org$month)

## [1] 1 2 3 4

unique(locations_org$hourtype)

## [1] "night"     "morning"   "afternoon" "evening"

unique(locations_org$hour)

##  [1]  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22
## [24] 23

head(locations_org)

## # A tibble: 6 x 7
##   month   day weekday   daytype  hour hourtype location
##   <dbl> <dbl> <chr>     <chr>   <dbl> <chr>    <chr>   
## 1     1     4 wednesday weekday     0 night    home    
## 2     1     4 wednesday weekday     1 night    home    
## 3     1     4 wednesday weekday     2 night    home    
## 4     1     4 wednesday weekday     3 night    home    
## 5     1     4 wednesday weekday     4 night    home    
## 6     1     4 wednesday weekday     5 night    home

locations = locations_org %>%
  select(daytype, hourtype, location)

str(locations)

## Classes 'spec_tbl_df', 'tbl_df', 'tbl' and 'data.frame': 2184 obs. of  3 variables:
##  $ daytype : chr  "weekday" "weekday" "weekday" "weekday" ...
##  $ hourtype: chr  "night" "night" "night" "night" ...
##  $ location: chr  "home" "home" "home" "home" ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   month = col_double(),
##   ..   day = col_double(),
##   ..   weekday = col_character(),
##   ..   daytype = col_character(),
##   ..   hour = col_double(),
##   ..   hourtype = col_character(),
##   ..   location = col_character()
##   .. )

# convert all columns to factor
locations = locations %>%
  mutate_all(factor)

str(locations)

## Classes 'spec_tbl_df', 'tbl_df', 'tbl' and 'data.frame': 2184 obs. of  3 variables:
##  $ daytype : Factor w/ 2 levels "weekday","weekend": 1 1 1 1 1 1 1 1 1 1 ...
##  $ hourtype: Factor w/ 4 levels "afternoon","evening",..: 4 4 4 4 4 4 3 3 3 3 ...
##  $ location: Factor w/ 7 levels "appointment",..: 3 3 3 3 3 3 3 3 3 4 ...

where9am = locations_org %>%
  filter(hour == 9) %>%
  select(daytype, location)

dim(where9am)

## [1] 91  2

23 Computing probabilities

The where9am data frame contains 91 days (thirteen weeks) worth of data in which Brett recorded his location at 9am each day as well as whether the daytype was a weekend or weekday.

Using the conditional probability formula below, you can compute the probability that Brett is working in the office, given that it is a weekday.

P(A|B) = P(A and B)/P(B)

Calculations like these are the basis of the Naive Bayes destination prediction model you’ll develop in later exercises.

head(where9am)

## # A tibble: 6 x 2
##   daytype location
##   <chr>   <chr>   
## 1 weekday office  
## 2 weekday office  
## 3 weekday office  
## 4 weekend home    
## 5 weekend home    
## 6 weekday campus

Find P(office) using nrow() and subset() to count rows in the dataset and save the result as p_A.

Find P(weekday), using nrow() and subset() again, and save the result as p_B.

Use nrow() and subset() a final time to find P(office and weekday). Save the result as p_AB.

Compute P(office | weekday) and save the result as p_A_given_B.

Print the value of p_A_given_B

# Using dplyr
atOffice = where9am %>% filter(location == "office")
isWeekday = where9am %>% filter(daytype == "weekday")
atOffice_isWeekday =where9am %>% filter(location == "office", daytype == "weekday")

head(atOffice)

## # A tibble: 6 x 2
##   daytype location
##   <chr>   <chr>   
## 1 weekday office  
## 2 weekday office  
## 3 weekday office  
## 4 weekday office  
## 5 weekday office  
## 6 weekday office

head(isWeekday)

## # A tibble: 6 x 2
##   daytype location   
##   <chr>   <chr>      
## 1 weekday office     
## 2 weekday office     
## 3 weekday office     
## 4 weekday campus     
## 5 weekday home       
## 6 weekday appointment

head(atOffice_isWeekday)

## # A tibble: 6 x 2
##   daytype location
##   <chr>   <chr>   
## 1 weekday office  
## 2 weekday office  
## 3 weekday office  
## 4 weekday office  
## 5 weekday office  
## 6 weekday office

(p_A = nrow(atOffice)/nrow(where9am))

## [1] 0.4285714

(p_B = nrow(isWeekday)/nrow(where9am))

## [1] 0.7142857

(p_AB = nrow(atOffice_isWeekday)/nrow(where9am))

## [1] 0.4285714

(p_A_given_B = p_AB/p_B)

## [1] 0.6

# using subscript
p_A = nrow(where9am[which(where9am$location == "office"),])/nrow(where9am)
p_A

## [1] 0.4285714

p_B = nrow(where9am[which(where9am$daytype == "weekday"),])/nrow(where9am)
p_B

## [1] 0.7142857

p_AB = nrow(where9am[which( where9am$location == "office" & 
                              where9am$daytype == "weekday"),])/nrow(where9am)
p_AB

## [1] 0.4285714

(p_A_given_B = p_AB/p_B)

## [1] 0.6

Great work! In a lot of cases, calculating probabilities is as simple as counting.

24 Understanding dependent events

In the previous exercise, you found that there is a 55% chance Brett is in the office at 9am given that it is a weekday. On the other hand, if Brett is never in the office on a weekend, which of the following is/are true?

Possible Answers P(office and weekend) = 0.

P(office | weekend) = 0.

Brett’s location is dependent on the day of the week.

All of the above. (answer)

Correct! Because the events do not overlap, knowing that one occurred tells you much about the status of the other.

A simple Naive Bayes location model The previous exercises showed that the probability that Brett is at work or at home at 9am is highly dependent on whether it is the weekend or a weekday.

To see this finding in action, use the where9am data frame to build a Naive Bayes model on the same data.

You can then use this model to predict the future: where does the model think that Brett will be at 9am on Thursday and at 9am on Saturday?

The dataframe where9am is available in your workspace. This dataset contains information about Brett’s location at 9am on different days.

Load the naivebayes package.

Use naive_bayes() with a formula like y ~ x to build a model of location as a function of daytype. Forecast the Thursday 9am location using predict() with the thursday9am object as the newdata argument. Do the same for predicting the saturday9am location

#install.packages("naivebayes")
library(naivebayes)

locmodel <- naive_bayes(location ~ daytype, data = where9am)

locmodel 

## ===================== Naive Bayes ===================== 
## Call: 
## naive_bayes.formula(formula = location ~ daytype, data = where9am)
## 
## A priori probabilities: 
## 
## appointment      campus        home      office 
##  0.01098901  0.10989011  0.45054945  0.42857143 
## 
## Tables: 
##          
## daytype   appointment    campus      home    office
##   weekday   1.0000000 1.0000000 0.3658537 1.0000000
##   weekend   0.0000000 0.0000000 0.6341463 0.0000000

names(where9am)

## [1] "daytype"  "location"

(thursday9am = data.frame(daytype = "weekday"))

##   daytype
## 1 weekday

saturday9am = data.frame(daytype = "weekend")
saturday9am

##   daytype
## 1 weekend

# making predictions with Naive Bayes
(predict(locmodel,  thursday9am))

## [1] office
## Levels: appointment campus home office

(predict(locmodel, saturday9am))

## [1] office
## Levels: appointment campus home office

This is incorrect

Answer should be as below

predict(locmodel, saturday9am)

## [1] office
## Levels: appointment campus home office

# However, below gives correct answer
(predict(locmodel, "weekend"))

## [1] home
## Levels: appointment campus home office

Awesome job! Not surprisingly, Brett is most likely at the office at 9am on a Thursday, but at home at the same time on a Saturday!

25 Examining ‘raw’ probabilities

The naivebayes package offers several ways to peek inside a Naive Bayes model.

Typing the name of the model object provides the a priori (overall) and conditional probabilities of each of the model’s predictors. If one were so inclined, you might use these for calculating posterior (predicted) probabilities by hand.

Alternatively, R will compute the posterior probabilities for you if the type = ‘prob’ parameter is supplied to the predict() function.

Using these methods, examine how the model’s predicted 9am location probability varies from day-to-day.

The model locmodel that you fit in the previous exercise is in your workspace.

Print the locmodel object to the console to view the computed a priori and conditional probabilities.

Use the predict() function similarly to the previous exercise, but with type = ‘prob’ to see the predicted probabilities for Thursday at 9am.

Compare these to the predicted probabilities for Saturday at 9am.

(predict(locmodel,  thursday9am, type = "prob"))

##      appointment    campus      home office
## [1,]  0.01538462 0.1538462 0.2307692    0.6

(predict(locmodel,  "weekend", type = "prob"))

##      appointment    campus      home    office
## [1,]  0.01098901 0.1098901 0.4505495 0.4285714

Should be :

appointment campus home office

[1,] 0 0 1 0

Fantastic! Did you notice the predicted probability of Brett being at the office on a Saturday is zero?

26 Understanding independence

Understanding the idea of event independence will become important as you learn more about how ‘naive’ Bayes got its name. Which of the following is true about independent events?

Possible Answers The events cannot occur at the same time.

A Venn diagram will always show no intersection.

Knowing the outcome of one event does not help predict the other. (answer)

At least one of the events is completely random.

Yes! One event is independent of another if knowing one doesn’t give you information about how likely the other is. For example, knowing if it’s raining in New York doesn’t help you predict the weather in San Francisco. The weather events in the two cities are independent of each other.

27 Understanding NB’s ‘naiivety’

• in the previous exercises we built a simple NB’s mode
• that used historic location data to predict my future location
• to build a more sophisticated model you may add additional data points to help inform the estimated probability of my location
• but until now we have only considered conditional when a single event predicts another
• adding more predictors complicates matters and that is the reason why this method is called naive

28 The challenge of multiple predictors

• with a single predictor conditional probability is based on the overlap between two events
• as the venn diagram in below image illustrates

NB Multi Venn

• as we start adding more events, the venn diagram can start to look a bit messy
• in the image we can see with three events
• imagine it with dozens or more
• and as confusing as this looks to us
• for a number of reasons it also becomes more inefficient for a computer to calculate the overlap

A ‘naive’ simplification

• instead the ‘naive’ based algorithm uses a simplification
• a shortcut to approximate the conditional probability we hope to compute
• rather than treating the problem as an intersection of all the related events the algorithm makes a so called naive assumption of the data
• specifically it assumes that the events are independent
• when events are independent the joint probability can be computed by multiplying the individual probabilities
• therefore under this naive assumption, the algorithm does not need to observe, all of the possible intersections in the full venn diagram
• instead it simply multiplies the probabilities from a series of much simpler intersections
• researches have found that although the naive assumptions is rarely true in practise the NB’s model still performs admirably on many real world tasks
• so there is little need to worry about a potential downside

29 An ‘infrequent’ problem

• there is one other potential issue to be aware of when building a naive bayes model
• suppose you have a set of predictors changed together under the naive assumption
• suppose further that one of those events has never been observed previously in combination with the outcome
• for instance I may never have gone to work on a weekend
• I may do this someday in the future, I just haven’t done so before
• in this case the venn diagram for work on weekend has no overlap

NB Infrequent.

• the joint probability of these two events is zero
• and whenever zero is multipled in a chain, the entire sequence becomes zero
• for this reason the weekend event has veto power over the entire prediction
• no matter how overwhelming the rest of the events, any predited probability of a work on a weekend will always be zero
• the solution to this problem is adding a small number, usually 1 to each of the events and outcome combination to eliminate this veto problem
• this is called Laplace correction or Laplace estimator

The Laplace correction {la pluz}

NB Laplace.

• after adding this correction, each venn diagram now has at least a small bit of overlap
• there is no longer any joint probability of zero
• as a result there will be at least some predicted probability for every future outcome, even if has never been seen before
• let’s build a more sophisticated NB model

30 Who are you calling naive?

The Naive Bayes algorithm got its name because it makes a ‘naive’ assumption about event independence.

What is the purpose of making this assumption?

Possible Answers Independent events can never have a joint probability of zero.

The joint probability calculation is simpler for independent events. (answer)

Conditional probability is undefined for dependent events.

Dependent events cannot be used to make predictions.

Yes! The joint probability of independent events can be computed much more simply by multiplying their individual probabilities.

31 A more sophisticated location model

The locations dataset records Brett’s location every hour for 13 weeks. Each hour, the tracking information includes the daytype (weekend or weekday) as well as the hourtype (morning, afternoon, evening, or night).

Using this data, build a more sophisticated model to see how Brett’s predicted location not only varies by the day of week but also by the time of day.

The dataset locations is already loaded in your workspace.

Use the R formula interface to build a model where location depends on both daytype and hourtype. Recall that the function naive_bayes() takes 2 arguments: formula and data.

Predict Brett’s location on a weekday afternoon using the dataframe weekday_afternoon and the predict() function.

Do the same for a weekday_evening

names(locations)

## [1] "daytype"  "hourtype" "location"

locmodel = naive_bayes(location ~ daytype + hourtype ,data = locations )

weekday_afternoon = data.frame(daytype = "weekday",
                               hourtype = "afternoon",
                               location = "office")

weekday_evening = data.frame(daytype = "weekday",
                               hourtype = "evening",
                               location = "home")

predict(locmodel, weekday_afternoon)

## [1] office
## Levels: appointment campus home office restaurant store theater

predict(locmodel, weekday_evening)

## [1] office
## Levels: appointment campus home office restaurant store theater

This is incorrect, should be as below

[1] home Levels: appointment campus home office restaurant store theater

Great job! Your Naive Bayes model forecasts that Brett will be at the office on a weekday afternoon and at home in the evening.

weekday_afternoon daytype hourtype location 13 weekday afternoon office weekday_evening daytype hourtype location 19 weekday evening home

32 Preparing for unforeseen circumstances

While Brett was tracking his location over 13 weeks, he never went into the office during the weekend. Consequently, the joint probability of P(office and weekend) = 0.

Explore how this impacts the predicted probability that Brett may go to work on the weekend in the future. Additionally, you can see how using the Laplace correction will allow a small chance for these types of unforeseen circumstances.

The model locmodel is already in your workspace, along with the dataframe weekend_afternoon.

Use the locmodel to output predicted probabilities for a weekend afternoon by using the predict() function. Remember to set the type argument.

Create a new naive Bayes model with the Laplace smoothing parameter set to 1. You can do this by setting the laplace argument in your call to naive_bayes(). Save this as locmodel2.

See how the new predicted probabilities compare by using the predict() function on your new model.

#rm(weekend_afternoon)

weekend_afternoon = locations %>%
  filter(daytype == "weekend", hourtype == "afternoon", location == "home") %>%
  head(1)

weekend_afternoon

## # A tibble: 1 x 3
##   daytype hourtype  location
##   <fct>   <fct>     <fct>   
## 1 weekend afternoon home

str(weekend_afternoon)

## Classes 'tbl_df', 'tbl' and 'data.frame':    1 obs. of  3 variables:
##  $ daytype : Factor w/ 2 levels "weekday","weekend": 2
##  $ hourtype: Factor w/ 4 levels "afternoon","evening",..: 1
##  $ location: Factor w/ 7 levels "appointment",..: 3

predict(locmodel, weekend_afternoon, type = "prob")

##      appointment campus      home office restaurant      store theater
## [1,]  0.02472535      0 0.8472217      0  0.1115693 0.01648357       0

locmodel2 = naive_bayes(location ~ daytype + hourtype, data = locations, laplace = 1)

predict(locmodel2, weekend_afternoon, type = "prob")

##      appointment      campus      home      office restaurant      store
## [1,]  0.01107985 0.005752078 0.8527053 0.008023444  0.1032598 0.01608175
##          theater
## [1,] 0.003097769

Fantastic work! Adding the Laplace correction allows for the small chance that Brett might go to the office on the weekend in the future.

Understanding the Laplace correction By default, the naive_bayes() function in the naivebayes package does not use the Laplace correction. What is the risk of leaving this parameter unset?

Possible Answers Some potential outcomes may be predicted to be impossible. (answer)

The algorithm may have a divide by zero error.

Naive Bayes will ignore features with zero values.

The model may not estimate probabilities for some cases.

Correct! The small probability added to every outcome ensures that they are all possible even if never previously observed.

Applying Naive Bayes to other problems

• smart phone destinations suggestions are a very specific application of Naive Bayes
• but the algorithm can be used for many other types of problems
• NB’s tends to work well on problems where the information from multiple attributes needs to be considered simultaneously and then evaluated as a whole
• the process is somewhat analogous to how a medical doctor might evaluate symtoms and test results to make a final diagnosis
• historically NB’s has also been frequently used for classifying text data
• like identifying whether or not an email is spam
• now I’ll present som of the challenges you may encountered when applying the algorithm to other classification tasks

How Naive Bayes uses data

NB Data.

• consider the fact that NB’s makes predictions by computing conditional probabilities of events and outcomes
• beginning with a tabular dataset it builds frequency tables that count the number of times each event overlaps with outcome of interest {table(locations\(daytype, locations\)location)}
• the probabilities are then multiplied naively in the chain of all the events
• the consequence of this approach is that each of the predictors used in a NB’s model, typically comprises of a set of categories
• numeric properties like age or time of day are difficult to use for NB’s as is without knowing more about the properties of the data
• similarly unstructured text data also defies categorization
• thus it is generally necessary to prepare these types of data before using them with NB’s

Binning numeric data for Naive Bayes * a technique called binning is a simple technique for creating categories from numeric data * the idea is to divide a range of numbers into a series of sets called bins

NB Binning 1

• for instance you might divided numbers into bins based on percentiles
• by creating a category for bottom 25%, the next 25%, and so on
• perhaps a better approach is to group ranges of values into meaningful bins
• for instance you might group times into categories like afternoon and evening
• and temperature readings into values like hot, warm and cold
• you can use R’s data preparating functions to recode data this way

33 Preparing text data for Naive Bayes

• text documents are considered unstructured data because they do not conform to the typical table or spreadsheet format of most datasets
• a common process for adding structure to text data uses a model called bag of words
• the bag of words model does not consider word order, grammar or symantecs
• it simply creates an event for each word that appears in a collection of text documents

NB BagOfWords

• for example the bag of words for this document on Understanding Naive Bayes, would include events for words like ‘naive’ and ‘bayes’ and understanding
• in spreadsheet form this results in a wide table where rows are documents and the columns are words that may appear in the documents
• each spreadsheet cell indicates whether or not the word appeared in that document
• when the NB’s algorithm is applied to the bag of words it can estimate the probability of the outcome given the evidence provided by the words in the text
• for instance a document with the words viagra and presciption is more likely to be spam than a document with words naive and bayes
• NB’s model trained with bag of words can be very effective text classifiers
• you can learn more about this in Data Camps text mining class which will teach you how to apply R’s tm package to build datasets you can use with naive bayes

34 Handling numeric predictors

Numeric data is often binned before it is used with Naive Bayes. Which of these is not an example of binning?

ANSWER THE QUESTION

Possible Answers age values recoded as ‘child’ or ‘adult’ categories

geographic coordinates recoded into geographic regions (West, East, etc.)

test scores divided into four groups by percentile

income values standardized to follow a normal bell curve (answer)

Right! Transforming income values into a bell curve doesn’t create a set of categories.

35 Making binary predictions with regression

• if you’ve spent any time at all studying data science you are likely to have encountered regression analysis which is a branch of statistics interested in modeling numeric relationships within data
• regression methods are perhaps the single most common form of machine learning
• the technique can be adapted to virtually any type of problem in any domain
• here you’ll seee how a regression model can be used to classify a binary outcome
• later you’ll use what you’ve learned to predict whether or not someone will donate to charity
• a topic directly related to my own work as a fundraising data scientist

36 Introducing linear regression

• in its most basic form, regression involves predicting an outcome y using one of more predictor labelled as x variables
• the y variable is known as the dependent variable as it seems to depend on the xs
• suppose you plot a numeric y vs a numeric x term

Linear Regression 1

{linear positive regression line}

• the y might reflect something like income or life expectance while the x-axis could represent age or education

• linear regression involves fitting a straight line to this data

• that best captures the relationship between the x and the y terms

Regression for binary classification * but suppose you have a binary term outcome instead * something that can take 1 or 0 values * like donate or not donate * constructing a plot of y vs x, the points fall on two flat rows rather than spread across

Linear Regression Binary 1

• you can still apply a straight line to the data it doesn’t fit well

Linear Regression Binary 2

• additionally for some values of x, the model will predict values less than zero or greater than one
• this is obviously not ideal

37 Introducing logistic regression

• now imagine the same binary outcome but rather than attempting to model straight line we use a curve instead
• this is the idea behind logistic regression

Linear Regression Logistic 2

• a type of S shaped curve called the logistic function has the property that for any input value of x the output is always between 0 and 1, just like a probability
• the greater this probability the more likely the output is outcome labelled 1

Making predictions with logistic regression * in R the logistic regression uses the glm function

\[m = glm(y ~ x1 + x2 + x3, data = my_dataset, family = 'binomial')\]

• the family parameter defines the type of model you are building
• because glm can be used to do many different types of regression
• binomil informs R to perform logistics regression
• once the model has been built it can be used to estimate probabilities

\[prob = predict(m, test_dataset, type = response)\]

• type as response produces predicted probabilities which are easier to interpret than the default log odds values ???

\[pred = ifelse(prob > 0.50, 1, 0)\]

• to make predictions these probabilities must be converted into the outcome of interest using an ifelse step
• which predicts outcome as 1 if probability is greater than 50% and zero otherwise
• sometimes you may need to set this threshold higher or lower to make the model more or less aggressive

donors_org = read_csv("C:/shobha/R/DataCamp/dataFiles/CSV-files/donors.csv")

## Parsed with column specification:
## cols(
##   donated = col_double(),
##   veteran = col_double(),
##   bad_address = col_double(),
##   age = col_double(),
##   has_children = col_double(),
##   wealth_rating = col_double(),
##   interest_veterans = col_double(),
##   interest_religion = col_double(),
##   pet_owner = col_double(),
##   catalog_shopper = col_double(),
##   recency = col_character(),
##   frequency = col_character(),
##   money = col_character()
## )

dim(donors_org)

## [1] 93462    13

names(donors_org)

##  [1] "donated"           "veteran"           "bad_address"      
##  [4] "age"               "has_children"      "wealth_rating"    
##  [7] "interest_veterans" "interest_religion" "pet_owner"        
## [10] "catalog_shopper"   "recency"           "frequency"        
## [13] "money"

38 Building simple logistic regression models

The donors dataset contains 93,462 examples of people mailed in a fundraising solicitation for paralyzed military veterans. The donated column is 1 if the person made a donation in response to the mailing and 0 otherwise. This binary outcome will be the dependent variable for the logistic regression model.

The remaining columns are features of the prospective donors that may influence their donation behavior. These are the model’s independent variables.

When building a regression model, it it often helpful to form a hypothesis about which independent variables will be predictive of the dependent variable. The bad_address column, which is set to 1 for an invalid mailing address and 0 otherwise, seems like it might reduce the chances of a donation. Similarly, one might suspect that religious interest (interest_religion) and interest in veterans affairs (interest_veterans) would be associated with greater charitable giving.

In this exercise, you will use these three factors to create a simple model of donation behavior.

{We build the glm regression model for lable donated using three independent variables/features bad_address, interest_veteran, interest_religion. - Then we use the model to predict the probalities for donation and save as a new variable. - then we calculate the mean of actual donated variable/label - then we make binary prediction using the if else condition that if prob is greater than the average donated assign predicted value as 1 otherwise 0 - the accuracy is then deduced as mean of the observations where predicted and actual value of donation is the same}

The dataset donors is available in your workspace.

Examine donors using the str() function.

Count the number of occurrences of each level of the donated variable using the table() function.

Fit a logistic regression model using the formula interface and the three independent variables described above.

• Call glm() with the formula as its first argument and the dataframe as the data argument.

• Save the result as donation_model.

Summarize the model object with summary()

donors = donors_org
glimpse(donors)

## Observations: 93,462
## Variables: 13
## $ donated           <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ veteran           <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ bad_address       <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ age               <dbl> 60, 46, NA, 70, 78, NA, 38, NA, NA, 65, NA, ...
## $ has_children      <dbl> 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0,...
## $ wealth_rating     <dbl> 0, 3, 1, 2, 1, 0, 2, 3, 1, 0, 1, 2, 1, 0, 2,...
## $ interest_veterans <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ interest_religion <dbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ pet_owner         <dbl> 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ catalog_shopper   <dbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ recency           <chr> "CURRENT", "CURRENT", "CURRENT", "CURRENT", ...
## $ frequency         <chr> "FREQUENT", "FREQUENT", "FREQUENT", "FREQUEN...
## $ money             <chr> "MEDIUM", "HIGH", "MEDIUM", "MEDIUM", "MEDIU...

# Examine the dataset to identify potential independent variables
str(donors)

## Classes 'spec_tbl_df', 'tbl_df', 'tbl' and 'data.frame': 93462 obs. of  13 variables:
##  $ donated          : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ veteran          : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ bad_address      : num  0 0 0 0 0 0 0 0 0 0 ...
##  $ age              : num  60 46 NA 70 78 NA 38 NA NA 65 ...
##  $ has_children     : num  0 1 0 0 1 0 1 0 0 0 ...
##  $ wealth_rating    : num  0 3 1 2 1 0 2 3 1 0 ...
##  $ interest_veterans: num  0 0 0 0 0 0 0 0 0 0 ...
##  $ interest_religion: num  0 0 0 0 1 0 0 0 0 0 ...
##  $ pet_owner        : num  0 0 0 0 0 0 1 0 0 0 ...
##  $ catalog_shopper  : num  0 0 0 0 1 0 0 0 0 0 ...
##  $ recency          : chr  "CURRENT" "CURRENT" "CURRENT" "CURRENT" ...
##  $ frequency        : chr  "FREQUENT" "FREQUENT" "FREQUENT" "FREQUENT" ...
##  $ money            : chr  "MEDIUM" "HIGH" "MEDIUM" "MEDIUM" ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   donated = col_double(),
##   ..   veteran = col_double(),
##   ..   bad_address = col_double(),
##   ..   age = col_double(),
##   ..   has_children = col_double(),
##   ..   wealth_rating = col_double(),
##   ..   interest_veterans = col_double(),
##   ..   interest_religion = col_double(),
##   ..   pet_owner = col_double(),
##   ..   catalog_shopper = col_double(),
##   ..   recency = col_character(),
##   ..   frequency = col_character(),
##   ..   money = col_character()
##   .. )

# Explore the dependent variable
table(donors = donors$donated)

## donors
##     0     1 
## 88751  4711

table(donors = donors$donated, bad_address = donors$bad_address)

##       bad_address
## donors     0     1
##      0 87440  1311
##      1  4660    51

Bad address donated 51 and correct address donated 4660 which equals total donated 4711

table(donors = donors$donated, interest_veterans = donors$interest_veterans)

##       interest_veterans
## donors     0     1
##      0 78998  9753
##      1  4133   578

table(donors = donors$donated, interest_religion = donors$interest_religion)

##       interest_religion
## donors     0     1
##      0 80439  8312
##      1  4231   480

Interest in religion donated 480 and not donated 4231 which equals 4711

# Build the donation model
donation_model <- glm(donated ~ bad_address + interest_veterans + interest_religion, data = donors, family = "binomial")

# Summarize the model results
summary(donation_model)

## 
## Call:
## glm(formula = donated ~ bad_address + interest_veterans + interest_religion, 
##     family = "binomial", data = donors)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -0.3480  -0.3192  -0.3192  -0.3192   2.5678  
## 
## Coefficients:
##                   Estimate Std. Error  z value Pr(>|z|)    
## (Intercept)       -2.95139    0.01652 -178.664   <2e-16 ***
## bad_address       -0.30780    0.14348   -2.145   0.0319 *  
## interest_veterans  0.11009    0.04676    2.354   0.0186 *  
## interest_religion  0.06724    0.05069    1.327   0.1847    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 37330  on 93461  degrees of freedom
## Residual deviance: 37316  on 93458  degrees of freedom
## AIC: 37324
## 
## Number of Fisher Scoring iterations: 5

Great work! With the model built, you can now use it to make predictions!

39 Making a binary prediction

In the previous exercise, you used the glm() function to build a logistic regression model of donor behavior. As with many of R’s machine learning methods, you can apply the predict() function to the model object to forecast future behavior. By default, predict() outputs predictions in terms of log odds unless type = ‘response’ is specified. This converts the log odds to probabilities.

Because a logistic regression model estimates the probability of the outcome, it is up to you to determine the threshold at which the probability implies action. One must balance the extremes of being too cautious versus being too aggressive. For example, if you were to solicit only the people with a 99% or greater donation probability, you may miss out on many people with lower estimated probabilities that still choose to donate. This balance is particularly important to consider for severely imbalanced outcomes, such as in this dataset where donations are relatively rare.

The dataset donors and the model donation_model are already loaded in your workspace.

Use the predict() function to estimate each person’s donation probability. Use the type argument to get probabilities. Assign the predictions to a new column called donation_prob.

Find the actual probability that an average person would donate by passing the mean() function the appropriate column of the donors dataframe.

Use ifelse() to predict a donation if their predicted donation probability is greater than average. Assign the predictions to a new column called donation_pred.

Use the mean() function to calculate the model’s accuracy.

# Estimate the donation probability
donors$donation_prob <- predict(donation_model, type = "response")
head(donors$donation_prob)

## [1] 0.04967101 0.04967101 0.04967101 0.04967101 0.05294280 0.04967101

# Find the donation probability of the average prospect
mean(donors$donated)

## [1] 0.05040551

# For no donation
mean(!donors$donated)

## [1] 0.9495945

# Predict a donation if probability of donation is greater than average (0.0504)
donors$donation_pred <- ifelse(donors$donation_prob > 0.0504, 1, 0)
head(donors$donation_pred)

## [1] 0 0 0 0 1 0

# Calculate the model's accuracy
mean(donors$donated == donors$donation_pred)

## [1] 0.794815

Nice work! With an accuracy of nearly 80%, the model seems to be doing its job. But is it too good to be true?

The limitations of accuracy In the previous exercise, you found that the logistic regression model made a correct prediction nearly 80% of the time. Despite this relatively high accuracy, the result is misleading due to the rarity of outcome being predicted.

The donors dataset is available in your workspace. What would the accuracy have been if a model had simply predicted ‘no donation’ for each person?

Possible Answers 80%

85%

90%

95% (answer - see working below)

Correct! With an accuracy of only 80%, the model is actually performing WORSE than if it were to predict non-donor for every record.

# For no donation
mean(!donors$donated)

## [1] 0.9495945

# Predict a donation if probability of donation is greater than average (0.0504)
donors$no_donation_pred <- ifelse(donors$donation_prob > 0.95, 0, 1)
head(donors$no_donation_pred)

## [1] 1 1 1 1 1 1

# Calculate the model's accuracy
mean(!donors$donated == donors$no_donation_pred)

## [1] 0.9495945

40 Model performance tradeoffs

• as the previous exercise illustrated rare events create challenges for classification models
• when one outcome is very rare, predicting the opposite can result in very hight accuracy
• this was the case in the donations dataset, because only about 5% of people were donors predicting non-donation resulted in an overall accuracy of 95% but an accuracy of zero on the outcome that mattered the most, the donations ??
• in cases like these it may be necessary to sacrifice a bit of overall accuracy in order to better target an outcome of interest

41 Understanding ROC curves

• a visualization called ROC curve, provides a way to better understand a models ability to distinguish between positive and negative prediction
• the outcome interest vs all the others
• to understand this better imagine you’re working on a project where the positive outcome is x and the negative outcome is o (o not zero)
• the classifier is trying to distinguish between these two groups
• if the classifier is poor the xs an os will remain very mixed as show in below image

Linear Regression ROC 1

• the ROC curve depicts the relationship between the percentage of positive examples as it relates to the percentage of other outcomes

Linear Regression ROC 3

• the ROC curve makes a diagonal line showing that proportion of interesting examples rises evenly with the proportion of negative examples
• on the other hand suppose we have machine learning model that is able to sort the examples of interest so that they appear to the front of the dataset
• the outcomes might be arranges as shown in below image

Linear Regression ROC 2

• with more xs on the left than on the right
• when the ROC curve is drawn for this arrangement, it is no longer along the diagonal because the model is able to identify several positive examples for each negative example it accidently prioritised

Linear Regression ROC 4

42 Area under the ROC curve (AUC)

• the diagonal line is the baseline performance for a very poor model
• the further another curve is away from this line the better it is performing
• conversely, a model that is very close to the diagonal line is not performing very well at all
• to quantify this performance a measurement called AUC or area under the curve is used

Linear Regression AUC 1

• the AUC literally measures the area under the ROC curve
• the baseline model which is no better than the random chance as in an AUC of 0.5, because it divides the 1 by 1 unit square perfectly in half
• a perfect model has an AUC of 1, with curve all the way in the upper left of the square
• most real world results are somewhere in between
• generally speaking the close the AUC is to 1 the better but there are some cases where the AUC can be misleading

43 Using AUC and ROC appropriately

• curves of varying shapes can have the same AUC value

Linear Regression AUC 2

• for this reason it is important to not only look at the AUC but also how the shape of each curve indicates how our model is performing across the range of predictions
• for example, one model may do extremely well at identifying a few easy cases at first
• but perform poorly on more difficult cases, another model may do just the opposite
• as the above figure shows both may end up with the same AUC
• ROC are an important tool for comparing models and selecting the best model for your project needs
• when used with a single model, it can help to visualize the tradeoff between true positives and false positives for the outcome of interest
• let’s plot ROC curves for donation dataset, to see how well the model is really performing

44 Calculating ROC Curves and AUC

The previous exercises have demonstrated that accuracy is a very misleading
measure of model performance on imbalanced datasets. Graphing the model’s
performance better illustrates the tradeoff between a model that is overly agressive and one that is overly passive.

In this exercise you will create a ROC curve and compute the area under the curve (AUC) to evaluate the logistic regression model of donations you built earlier.

The dataset donors with the column of predicted probabilities, donation_prob, is already loaded in your workspace.

Load the pROC package.

Create a ROC curve with roc() and the columns of actual and predicted donations.

Store the result as ROC.

Use plot() to draw the ROC object. Specify col = ‘blue’ to color the curve blue.

Compute the area under the curve with auc().

# Load the pROC package
# install.packages("pROC")
library(pROC)

## Warning: package 'pROC' was built under R version 3.5.3

## Type 'citation("pROC")' for a citation.

## 
## Attaching package: 'pROC'

## The following objects are masked from 'package:stats':
## 
##     cov, smooth, var

# Create a ROC curve
ROC <- roc(donors$donated, donors$donation_prob)

# Plot the ROC curve
plot(ROC, col = "blue")

# Calculate the area under the curve (AUC)
auc(ROC)

## Area under the curve: 0.5102

Area under the curve: 0.5102

Awesome job! Based on this visualization, the model isn’t doing much better than baseline- a model doing nothing but making predictions at random.

While comparing AUCs for ROCs

• Correct! When AUC values are very close, it’s important to know more about how the model will be used.

45 Dummy variables, missing data, and interactions

• all of the predictors used in our regression analysis must be numeric
• this means all categorical data must be represented as number
• missing data also posses as a missing value cannot be used to make predictions
• now you will learn tips for preparing these types of data to be used in logistics linear regression model
• you will also learn how to model the interactions among predictors, an important step in building more powerful predictive models

46 Dummy coding categorical data

• in chapter one you learned about dummy coding which creates a set of binary 1 0 variables that represent each category except one that serves as the reference group
• dummy coding is the most common method for handling categorical data for logistic regression
• the glm function will automatically dummy code any factor type variables used in the model
• simply aplly the factor function to the data as below

47 create gender factor

my_data$gender = factor(my_data$gender,
                    levels = c(0,1,2),
                    labels = c('Male','Female','Other'))

• keep in mind that you might run into a case where categorical feature is represented with numbers, such as 1,2,3 for Hot Warm and Cold
• even in this case it may be advisable to convert this to a factor
• this allows each category to have a unique impact on the outcome

48 Imputing missing data

• by default the regression model will exclude any observation with missing values on its predictors
• this may not be a big deal on small amounts of missing data but can very quickly become a much larger problem
• with a categorical missing data a missing value can be treated like any other category
• you might construct category for male female other and missing
• but when a numeric value is missing the solution is less clear
• one potential solution is using a technique called imputation
• this fills or imputes the missing value with a guess about what the value may be
• a very simple strategy is called mean imputation, which as you might expect, imputes the average
• because records having missing data may differ systematically from those without, a binary 1,0 missing value indicator can be added to model the fact that a value was imputed
• sometimes this becomes the model’s most important predictor
• it is important to note tha although this strategy is okay for simple predictive models, it is not okay for every regression application
• more sophisticated forms of imputation use models to predict the missing data based on the non-missing values

49 Interaction effects

• an interaction effect considers the fact that two predictors when combined may have a different impact on the outcome than the sum of their individual components
• the combination may strengthen weaken or completely eliminate the impact of the individual predictors
• For example, obesity and smoking both known to be harmful to one’s health but put together they may be even more harmful

Linear Regression Interact

• alternatively, two predictors may be harmful when applied separately, but when combined they neutralise suppress or nullify each other

Linear Regression Interact

• being able to model these combinations is important for creating the best predictive models
• as illustrated below, the R interface formula uses the multiplication symbol to indicate the interaction between the two predictors

50 interaction of obesity and smoking

glm(disease ~ obesity * smoking,
            data = health,
            factor = 'binomial')

• the resulting model will include a term for each of the individual components as well as the combined effect

• let’s apply dummy coding, missing value imputation and interaction effects to build a stronger donation model

51 Coding categorical features

Sometimes a dataset contains numeric values that represent a categorical feature.

In the donors dataset, wealth_rating uses numbers to indicate the donor’s wealth level:

0 = Unknown 1 = Low 2 = Medium 3 = High

This exercise illustrates how to prepare this type of categorical feature and the examines its impact on a logistic regression model.

The dataframe donors is loaded in your workspace.

Create a factor from the numeric wealth_rating with labels as shown above by passing the factor() function the column you want to convert, the individual levels, and the labels.

Use relevel() to change the reference category to Medium. The first argument should be your factor column.

Build a logistic regression model using the column wealth_rating to predict donated and display the result with summary().

str(donors$wealth_rating)

##  num [1:93462] 0 3 1 2 1 0 2 3 1 0 ...

length(unique(donors$wealth_rating))

## [1] 4

unique(donors$wealth_rating)

## [1] 0 3 1 2

# convert wealth rating to factor variable
donors$wealth_rating = factor(donors$wealth_rating,
                              levels = c(0,1,2,3),
                              labels = c("Unknown", "Low", "Medium", "High"))

# create reference category for dummy coding
# Use relevel() to change reference category
donors$wealth_rating <- relevel(donors$wealth_rating, ref = "Medium")

# See how our factor coding impacts the model
summary(glm(donated ~ wealth_rating, 
            family = 'binomial',
            data = donors))

## 
## Call:
## glm(formula = donated ~ wealth_rating, family = "binomial", data = donors)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -0.3320  -0.3243  -0.3175  -0.3175   2.4582  
## 
## Coefficients:
##                      Estimate Std. Error z value Pr(>|z|)    
## (Intercept)          -2.91894    0.03614 -80.772   <2e-16 ***
## wealth_ratingUnknown -0.04373    0.04243  -1.031    0.303    
## wealth_ratingLow     -0.05245    0.05332  -0.984    0.325    
## wealth_ratingHigh     0.04804    0.04768   1.008    0.314    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 37330  on 93461  degrees of freedom
## Residual deviance: 37323  on 93458  degrees of freedom
## AIC: 37331
## 
## Number of Fisher Scoring iterations: 5

Great job! What would the model output have looked like if this variable had been left as a numeric column?

52 Handling missing data

Some of the prospective donors have missing age data. Unfortunately, R will exclude any cases with NA values when building a regression model.

One workaround is to replace, or impute, the missing values with an estimated value. After doing so, you may also create a missing data indicator to model the possibility that cases with missing data are different in some way from those without.

The dataframe donors is loaded in your workspace.

Use summary() on donors to find the average age of prospects with non-missing data.

Use ifelse() and the test is.na(donors$age) to impute the average (rounded to 2 decimal places) for cases with missing age.

Create a binary dummy variable named missing_age indicating the presence of missing data using another ifelse() call and the same test.

summary(donors$age)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    1.00   48.00   62.00   61.65   75.00   98.00   22546

avg_age = mean(donors$age, na.rm = TRUE)
avg_age

## [1] 61.64787

donors$imputed_age = ifelse(is.na(donors$age), 61.65, donors$age)

summary(donors$imputed_age)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00   52.00   61.65   61.65   72.00   98.00

donors$missing_age = ifelse(is.na(donors$age), 1, 0)

#cross check the new age variables
donors %>%
  filter(is.na(age)) %>%
  select(age, missing_age, imputed_age) %>%
  head()

## # A tibble: 6 x 3
##     age missing_age imputed_age
##   <dbl>       <dbl>       <dbl>
## 1    NA           1        61.6
## 2    NA           1        61.6
## 3    NA           1        61.6
## 4    NA           1        61.6
## 5    NA           1        61.6
## 6    NA           1        61.6

Super! This is one way to handle missing data, but be careful! Sometimes missing data has to be dealt with using more complicated methods.

53 Understanding missing value indicators

A missing value indicator provides a reminder that, before imputation, there was a missing value present on the record.

Why is it often useful to include this indicator as a predictor in the model?

ANSWER THE QUESTION

Possible Answers A missing value may represent a unique category by itself

There may be an important difference between records with and without missing data

Whatever caused the missing value may also be related to the outcome

All of the above (answer)

Yes! Sometimes a missing value says a great deal about the record it appeared on!

54 Building a more sophisticated model

One of the best predictors of future giving is a history of recent, frequent, and large gifts. In marketing terms, this is known as R/F/M:

Recency Frequency Money Donors that haven’t given both recently and frequently may be especially likely to give again; in other words, the combined impact of recency and frequency may be greater than the sum of the separate effects.

Because these predictors together have a greater impact on the dependent variable, their joint effect must be modeled as an interaction.

55 Building a more sophisticated model

One of the best predictors of future giving is a history of recent, frequent, and large gifts. In marketing terms, this is known as R/F/M:

Recency Frequency Money Donors that haven’t given both recently and frequently may be especially likely to give again; in other words, the combined impact of recency and frequency may be greater than the sum of the separate effects.

Because these predictors together have a greater impact on the dependent variable, their joint effect must be modeled as an interaction.

56 The donors dataset has been loaded for you.

Create a logistic regression model of donated as a function of money plus the interaction of recency and frequency. Use * to add the interaction term.

Examine the model’s summary() to confirm the interaction effect was added.

Save the model’s predicted probabilities as rfm_prob. Use the predict() function, and remember to set the type argument.

Plot a ROC curve by using the function roc(). Remember, this function takes the column of outcomes and the vector of predictions.

Compute the AUC for the new model with the function auc() and compare performance to the simpler model.

# Build a recency, frequency, and money (RFM) model
rfm_model <- glm(donated ~ money + recency*frequency,family = 'binomial', data = donors)

# Summarize the RFM model to see how the parameters were coded
summary(rfm_model)

## 
## Call:
## glm(formula = donated ~ money + recency * frequency, family = "binomial", 
##     data = donors)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -0.3696  -0.3696  -0.2895  -0.2895   2.7924  
## 
## Coefficients:
##                                   Estimate Std. Error z value Pr(>|z|)    
## (Intercept)                       -3.01142    0.04279 -70.375   <2e-16 ***
## moneyMEDIUM                        0.36186    0.04300   8.415   <2e-16 ***
## recencyLAPSED                     -0.86677    0.41434  -2.092   0.0364 *  
## frequencyINFREQUENT               -0.50148    0.03107 -16.143   <2e-16 ***
## recencyLAPSED:frequencyINFREQUENT  1.01787    0.51713   1.968   0.0490 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 37330  on 93461  degrees of freedom
## Residual deviance: 36938  on 93457  degrees of freedom
## AIC: 36948
## 
## Number of Fisher Scoring iterations: 6

# Compute predicted probabilities for the RFM model
rfm_prob <- predict(rfm_model, type = "response")

# Plot the ROC curve and find AUC for the new model
library(pROC)
ROC <- roc(donors$donated,rfm_prob)
plot(ROC, col = "red")

auc(ROC)

## Area under the curve: 0.5785

Great work! Based on the ROC curve, you’ve confirmed that past giving patterns are certainly predictive of future giving.

57 Automatic feature selection

• unlike some machine learning methods regression typically asks the human to classify the model’s predictors ahead of time
• thus each of the donation models you have built so far required a little bit of fund raising subject matter expertise to identify the variables that may be predictive of donations
• sometimes you may not have such insight ahead of time
• you may not know what all the predictor mean or you may have so many predictors that there is no easy way to sort through them all
• a process called automatic feature selection can be used here
• but you’ll soon see that with this great power comes great responsibility to apply it carefully

58 Stepwise regression

• involves building a regression model step by step evaluating each predictor to see which ones add value to the final model
• a procedure called backward deletion begins with a model containing all of the predictors

Linear Regression Stepwise 1

• it then checks to see what happens when each one of the predictors is removed from the model
• if removing a predictor does not substantially impact the model’s ability to predict the outcome, then it can be safely deleted
• at each step the predictor that impacts the model the least is removed assuming ofcourse that it has minimal impact
• this continues step by step until only important predictors remain
• the same idea applied in the other direction is called forward selection
• begining with a model contianing no predictors, it examines each potential predictor to see which one if any offers the greatest improvement to the models predictive power
• predictors are added step by step until no new predictors add substantial value to the model

Linear Regression Stepwise 1

and

Linear Regression Stepwise 1

• keep in mind that although the figures here show the same final model backward and forward stepwise, this is not always the case
• it is possible that the two come to completely different solutions about the most important predictors
• this is just one of the possible caveats of stepwise regression

59 Stepwise regression caveats

• not only can backward and forward selection create completely different models
• but neither is guaranteed to find the best possible model
• statisticians also raise concerns about the fact that stepwise model violates some of the principles that allow a regression model to explain data as well as predict
• ofcourse if you only care about predictive power this may not be a very big concern
• the use of stepwise doesn’t mean that the model’s predictions are worthless
• it simply means the model may over or under state the importance of some of the predictors
• perhaps most importantly feature selection methods like stepwise progression allow the model to be built in the absense of theory or even commonsense
• this might result in a model that seems counter intuitive in the real world
• it may be best to consider stepwise regression as just one tool for exploring potential models in the absense of another good starting point

Linear Regression Stepwise Caveats

60 The dangers of stepwise regression

In spite of its utility for feature selection, stepwise regression is not frequently used in disciplines outside of machine learning due to some important caveats. Which of these is NOT one of these concerns?

ANSWER THE QUESTION

Possible Answers

It is not guaranteed to find the best possible model

A stepwise model’s predictions can not be trusted (answer - this is not a concern)

The stepwise regression procedure violates some statistical assumptions

It can result in a model that makes little sense in the real world

Correct! Though stepwise regression is frowned upon, it may still be useful for building predictive models in the absence of another starting place.

Building a stepwise regression model In the absence of subject-matter expertise, stepwise regression can assist with the search for the most important predictors of the outcome of interest.

In this exercise, you will use a forward stepwise approach to add predictors to the model one-by-one until no additional benefit is seen.

The donors dataset has been loaded for you.

Use the R formula interface with glm() to specify the base model with no predictors. Set the explanatory variable equal to 1.

Use the R formula interface again with glm() to specify the model with all predictors.

Apply step() to these models to perform forward stepwise regression. Set the first argument to null_model and set direction = ‘forward.’ This might take a while (up to 10 or 15 seconds) as your computer has to fit quite a few different models to perform stepwise selection.

Create a vector of predicted probabilities using the predict() function.

Plot the ROC curve with roc() and plot() and compute the AUC of the stepwise model with auc().

# Specify a null model with no predictors
null_model <- glm(donated ~ 1, data = donors, family = "binomial")

# Specify the full model using all of the potential predictors
full_model <- glm(donated ~ ., data = donors, family = "binomial")

# Use a forward stepwise algorithm to build a parsimonious model
step_model <- step(null_model, 
                   scope = list(lower = null_model, upper = full_model), 
                   direction = "forward")

## Start:  AIC=37332.13
## donated ~ 1

## Warning in add1.glm(fit, scope$add, scale = scale, trace = trace, k = k, :
## using the 70916/93462 rows from a combined fit

##                     Df Deviance   AIC
## + frequency          1    28502 37122
## + money              1    28621 37241
## + has_children       1    28705 37326
## + age                1    28707 37328
## + imputed_age        1    28707 37328
## + wealth_rating      3    28704 37328
## + interest_veterans  1    28709 37330
## + donation_prob      1    28710 37330
## + donation_pred      1    28710 37330
## + catalog_shopper    1    28710 37330
## + pet_owner          1    28711 37331
## <none>                    28714 37332
## + interest_religion  1    28712 37333
## + recency            1    28713 37333
## + bad_address        1    28714 37334
## + veteran            1    28714 37334
## 
## Step:  AIC=37024.77
## donated ~ frequency

## Warning in add1.glm(fit, scope$add, scale = scale, trace = trace, k = k, :
## using the 70916/93462 rows from a combined fit

##                     Df Deviance   AIC
## + money              1    28441 36966
## + wealth_rating      3    28490 37019
## + has_children       1    28494 37019
## + donation_prob      1    28498 37023
## + interest_veterans  1    28498 37023
## + catalog_shopper    1    28499 37024
## + donation_pred      1    28499 37024
## + age                1    28499 37024
## + imputed_age        1    28499 37024
## + pet_owner          1    28499 37024
## <none>                    28502 37025
## + interest_religion  1    28501 37026
## + recency            1    28501 37026
## + bad_address        1    28502 37026
## + veteran            1    28502 37027
## 
## Step:  AIC=36949.71
## donated ~ frequency + money

## Warning in add1.glm(fit, scope$add, scale = scale, trace = trace, k = k, :
## using the 70916/93462 rows from a combined fit

##                     Df Deviance   AIC
## + wealth_rating      3    28427 36942
## + has_children       1    28432 36943
## + interest_veterans  1    28438 36948
## + donation_prob      1    28438 36949
## + catalog_shopper    1    28438 36949
## + donation_pred      1    28439 36949
## + age                1    28439 36949
## + imputed_age        1    28439 36949
## + pet_owner          1    28439 36949
## <none>                    28441 36950
## + interest_religion  1    28440 36951
## + recency            1    28441 36951
## + bad_address        1    28441 36951
## + veteran            1    28441 36952
## 
## Step:  AIC=36945.48
## donated ~ frequency + money + wealth_rating

## Warning in add1.glm(fit, scope$add, scale = scale, trace = trace, k = k, :
## using the 70916/93462 rows from a combined fit

##                     Df Deviance   AIC
## + has_children       1    28416 36937
## + age                1    28424 36944
## + imputed_age        1    28424 36944
## + interest_veterans  1    28424 36945
## + donation_prob      1    28424 36945
## + catalog_shopper    1    28425 36945
## + donation_pred      1    28425 36945
## <none>                    28427 36945
## + pet_owner          1    28425 36946
## + interest_religion  1    28426 36947
## + recency            1    28427 36947
## + bad_address        1    28427 36947
## + veteran            1    28427 36947
## 
## Step:  AIC=36938.4
## donated ~ frequency + money + wealth_rating + has_children

## Warning in add1.glm(fit, scope$add, scale = scale, trace = trace, k = k, :
## using the 70916/93462 rows from a combined fit

##                     Df Deviance   AIC
## + pet_owner          1    28413 36937
## + donation_prob      1    28413 36937
## + catalog_shopper    1    28413 36937
## + interest_veterans  1    28413 36937
## + donation_pred      1    28414 36938
## <none>                    28416 36938
## + interest_religion  1    28415 36939
## + age                1    28416 36940
## + imputed_age        1    28416 36940
## + recency            1    28416 36940
## + bad_address        1    28416 36940
## + veteran            1    28416 36940
## 
## Step:  AIC=36932.25
## donated ~ frequency + money + wealth_rating + has_children + 
##     pet_owner

## Warning in add1.glm(fit, scope$add, scale = scale, trace = trace, k = k, :
## using the 70916/93462 rows from a combined fit

##                     Df Deviance   AIC
## <none>                    28413 36932
## + donation_prob      1    28411 36932
## + interest_veterans  1    28411 36932
## + catalog_shopper    1    28412 36933
## + donation_pred      1    28412 36933
## + age                1    28412 36933
## + imputed_age        1    28412 36933
## + recency            1    28413 36934
## + interest_religion  1    28413 36934
## + bad_address        1    28413 36934
## + veteran            1    28413 36934

# Estimate the stepwise donation probability
step_prob <- predict(step_model, type = "response")

# Plot the ROC of the stepwise model
library(pROC)
ROC <- roc(donors$donated, step_prob)
plot(ROC, col = "red")

auc(ROC)

## Area under the curve: 0.5849

Area under the curve: 0.5849 Should be Area under the curve: 0.6006

Fantastic work! Despite the caveats of stepwise regression, it seems to have resulted in a relatively strong model!

61 Making decisions with trees

• sometimes a difficult or complex decision can be made simpler by breaking it down into a series of smaller decisions
• if you’re considering taking a new job offer you might define requirement for accepting the position
• does it offer a high enough salary, does it have a long commute or does it require long hours, does it provide free coffee
• classification trees also known as decision trees work much the same way
• they’re used to find a set of ifelse conditions that are helpful for taking action
• as you will see soon because decisions are easily understood without statistics they can be useful for business strategy
• especially in areas where transparency is needed, like loan application approval

62 A decsion tree model

• let’s start by considering a decision tree structure

Decision Tree Structure

• as you might expect it closely resembles the real world trees
• the goal is to model the relationship between the predictors and an outcome of interest
• beginning at the root node data flows through ifelse decisions nodes that split the data according to it’s attributes
• the branches indicate the potential choices and leaf nodes denote final decisions
• these are also known as terminal nodes because they terminate the decision making process
• to understand how a tree structure is built, let’s consider a business process like whether or not to provide someone a loan
• after an applicant fills out a form with personal information like income, credit history and loan purpose
• the bank must quickly decide whether or not the individual is likely to repay the debt
• using historical applicant data and loan outcomes a classification tree can be built to learn the criteria that are most predictive of future loan payment

63 Divide-and-conquer

• growing a decision the decision tree uses a process called divide and conquer
• because it attempts to divide the dataset into partitions with similar values for the outcome of interest
• for loan applications it needs to separate applicants that are likely to repay from those that are likely to default on the debt
• suppose the tree considers two aspects of the applicant
• the credit score and the requested loan amount
• this figure visualizes these characteristics in relation to whether the loan is repaid

Decision Tree Split 1

• to divide and conquer the algorithm looks for an initial split that creates the two most homogenous groups
• first it splits into groups of high and low credit scores then it divides and conquers again with another split creating groups for high and low requested amounts

Decision Tree Split 3

• each one of these splits results in an ifelse decision in a tree structure

Decision Tree Split 2

• if the credit score is low, it predicts loan default
• if the credit score is high, and the loan value is large it also predicts default
• otherwise it predicts repay
• obviously a decision tree built on actual learning data is likely to be more complex
• but this illustrates the basic process of how such a tree might be built
• you’ll learn more about it shortly
• for now lets ignore the implementation details to focus on putting the algorithm to work

64 Building trees in R

• there are several packages that can be used for building classification trees in R
• one of the most widely used is called rpart, for recursive partitioning, a synonym for divide-and-conquer

65 building a simple rpart classification tree

library(rpart)
m = rpart(outcome ~ loan_amount + credit_score, 
            data = loans,   
            method = class)

• the class parameter tells R to build a classification tree
• and like the othe ML methods you’ve seen before
• the predict function obtains teh predicted class values used for the test dataset

66 making predictions from an rpart tree

\[p = predict(m, testData, type = 'class')\]

67 Building a simple decision tree

The loans dataset contains 11,312 randomly-selected people who were applied for and later received loans from Lending Club, a US-based peer-to-peer lending company.

You will use a decision tree to try to learn patterns in the outcome of these loans (either repaid or default) based on the requested loan amount and credit score at the time of application.

Then, see how the tree’s predictions differ for an applicant with good credit versus one with bad credit.

loans_org = read_csv("C:/shobha/R/DataCamp/dataFiles/CSV-files/loans.csv")

## Parsed with column specification:
## cols(
##   keep = col_double(),
##   rand = col_double(),
##   default = col_double(),
##   loan_amount = col_character(),
##   emp_length = col_character(),
##   home_ownership = col_character(),
##   income = col_character(),
##   loan_purpose = col_character(),
##   debt_to_income = col_character(),
##   credit_score = col_character(),
##   recent_inquiry = col_character(),
##   delinquent = col_character(),
##   credit_accounts = col_character(),
##   bad_public_record = col_character(),
##   credit_utilization = col_character(),
##   past_bankrupt = col_character()
## )

dim(loans_org)

## [1] 39732    16

glimpse(loans_org)

## Observations: 39,732
## Variables: 16
## $ keep               <dbl> 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1...
## $ rand               <dbl> 0.13046525, 0.99815098, 0.62827558, 0.25240...
## $ default            <dbl> 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1...
## $ loan_amount        <chr> "LOW", "LOW", "LOW", "MEDIUM", "LOW", "LOW"...
## $ emp_length         <chr> "10+ years", "< 2 years", "10+ years", "10+...
## $ home_ownership     <chr> "RENT", "RENT", "RENT", "RENT", "RENT", "RE...
## $ income             <chr> "LOW", "LOW", "LOW", "MEDIUM", "HIGH", "LOW...
## $ loan_purpose       <chr> "credit_card", "car", "small_business", "ot...
## $ debt_to_income     <chr> "HIGH", "LOW", "AVERAGE", "HIGH", "AVERAGE"...
## $ credit_score       <chr> "AVERAGE", "AVERAGE", "AVERAGE", "AVERAGE",...
## $ recent_inquiry     <chr> "YES", "YES", "YES", "YES", "NO", "YES", "Y...
## $ delinquent         <chr> "NEVER", "NEVER", "NEVER", "MORE THAN 2 YEA...
## $ credit_accounts    <chr> "FEW", "FEW", "FEW", "AVERAGE", "MANY", "AV...
## $ bad_public_record  <chr> "NO", "NO", "NO", "NO", "NO", "NO", "NO", "...
## $ credit_utilization <chr> "HIGH", "LOW", "HIGH", "LOW", "MEDIUM", "ME...
## $ past_bankrupt      <chr> "NO", "NO", "NO", "NO", "NO", "NO", "NO", "...

names(loans_org)

##  [1] "keep"               "rand"               "default"           
##  [4] "loan_amount"        "emp_length"         "home_ownership"    
##  [7] "income"             "loan_purpose"       "debt_to_income"    
## [10] "credit_score"       "recent_inquiry"     "delinquent"        
## [13] "credit_accounts"    "bad_public_record"  "credit_utilization"
## [16] "past_bankrupt"

dim(loans_org)

## [1] 39732    16

head(loans_org$outcome)

## Warning: Unknown or uninitialised column: 'outcome'.

## NULL

loans = loans_org %>% select(-keep, -rand)

dim(loans)

## [1] 39732    14

names(loans)

##  [1] "default"            "loan_amount"        "emp_length"        
##  [4] "home_ownership"     "income"             "loan_purpose"      
##  [7] "debt_to_income"     "credit_score"       "recent_inquiry"    
## [10] "delinquent"         "credit_accounts"    "bad_public_record" 
## [13] "credit_utilization" "past_bankrupt"

head(loans$default)

## [1] 0 1 0 0 0 0

loans = loans %>%
  mutate(outcome = factor(ifelse((loans$default==1), "repaid", "default")))

head(loans$default)

## [1] 0 1 0 0 0 0

head(loans$outcome)

## [1] default repaid  default default default default
## Levels: default repaid

loans = loans %>%
  select(-default)

dim(loans)

## [1] 39732    14

The dataset loans is already in your workspace.

Load the rpart package.

Fit a decision tree model with the function rpart().

Supply the R formula that specifies outcome as a function of loan_amount and credit_score as the first argument.

Leave the control argument alone for now. (You’ll learn more about that later!)

Use predict() with the resulting loan model to predict the outcome for the good_credit applicant. Use the type argument to predict the ‘class’ of the outcome.

Do the same for the bad_credit applicant.

str(loans$bad_public_record)

##  chr [1:39732] "NO" "NO" "NO" "NO" "NO" "NO" "NO" "NO" "NO" "NO" "NO" ...

table(loans$bad_public_record)

## 
##    NO   YES 
## 37613  2119

head(loans,1)

## # A tibble: 1 x 14
##   loan_amount emp_length home_ownership income loan_purpose debt_to_income
##   <chr>       <chr>      <chr>          <chr>  <chr>        <chr>         
## 1 LOW         10+ years  RENT           LOW    credit_card  HIGH          
## # ... with 8 more variables: credit_score <chr>, recent_inquiry <chr>,
## #   delinquent <chr>, credit_accounts <chr>, bad_public_record <chr>,
## #   credit_utilization <chr>, past_bankrupt <chr>, outcome <fct>

good_credit = data.frame("loan_amount" = "LOW",
                         "emp_length" = "10+ years",
                         "home_ownership" = "MORTGAGE",
                         "income" = "HIGH",
                         "loan_purpose" = "major_purchase",
                         "debt_to_income" = "AVERAGE",
                         "credit_score" = "HIGH",
                         "recent_inquiry" = "NO",
                         "delinquent" = "NEVER",
                         "credit_accounts" = "MANY",
                         "bad_public_record" = "NO",
                         "credit_utilization" = "LOW",
                         "past_bankrupt" = "NO",
                         "outcome" = "repaid" )

good_credit

##   loan_amount emp_length home_ownership income   loan_purpose
## 1         LOW  10+ years       MORTGAGE   HIGH major_purchase
##   debt_to_income credit_score recent_inquiry delinquent credit_accounts
## 1        AVERAGE         HIGH             NO      NEVER            MANY
##   bad_public_record credit_utilization past_bankrupt outcome
## 1                NO                LOW            NO  repaid

bad_credit = data.frame("loan_amount" = "LOW",
                        "emp_length" = "6 - 9 years",
                        "home_ownership" = "RENT",
                        "income" = "HIGH",
                        "loan_purpose" = "car",
                        "debt_to_income" = "LOW",
                        "credit_score" = "LOW",
                        "recent_inquiry" = "YES",
                        "delinquent" = "NEVER",
                        "credit_accounts" = "FEW",
                        "bad_public_record" = "NO",
                        "credit_utilization" = "HIGH",
                        "past_bankrupt" = "NO",
                        "outcome" = "repaid" )

bad_credit

##   loan_amount  emp_length home_ownership income loan_purpose
## 1         LOW 6 - 9 years           RENT   HIGH          car
##   debt_to_income credit_score recent_inquiry delinquent credit_accounts
## 1            LOW          LOW            YES      NEVER             FEW
##   bad_public_record credit_utilization past_bankrupt outcome
## 1                NO               HIGH            NO  repaid

# Load the rpart package
library(rpart)

# Build a lending model predicting loan outcome versus loan amount and credit score
loan_model <- rpart(outcome ~ loan_amount + credit_score, data = loans, method = "class", control = rpart.control(cp = 0))

# Make a prediction for someone with good credit
predict(loan_model, good_credit, type = "class")

## [1] default
## Levels: default repaid

# Make a prediction for someone with bad credit
predict(loan_model, bad_credit, type = "class")

## [1] default
## Levels: default repaid

Great job! Growing a decision tree is certainly faster than growing a real tree!

68 Visualizing classification trees

Due to government rules to prevent illegal discrimination, lenders are required to explain why a loan application was rejected.

The structure of classification trees can be depicted visually, which helps to understand how the tree makes its decisions.

The model loan_model that you fit in the last exercise is in your workspace.

Type loan_model to see a text representation of the classification tree.

Load the rpart.plot package.

Apply the rpart.plot() function to the loan model to visualize the tree.

See how changing other plotting parameters impacts the visualization by running the supplied command.

# Examine the loan_model object
loan_model

## n= 39732 
## 
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
## 
## 1) root 39732 5654 default (0.8576966 0.1423034) *

"
first build model using good_credit and bad_credit
"

## [1] "\nfirst build model using good_credit and bad_credit\n"

# Load the rpart.plot package
#install.packages("rpart.plot")
library(rpart.plot)

## Warning: package 'rpart.plot' was built under R version 3.5.3

# Plot the loan_model with default settings
rpart.plot(loan_model)

# Plot the loan_model with customized settings
rpart.plot(loan_model, 
           type = 3, 
           box.palette = c("red", "green"), 
           fallen.leaves = TRUE)

Awesome! What do you think of the fancy visualization?

69 Growing larger calssification trees

• starting from a seed real world trees need the proper combination of soil water air and light to grow
• just like understanding these principles helps you to become a better gardner knowing a bit more about the growing conditions of decision trees will help you to build more robust classification models
• in this lesson you’ll learn more about how trees grow, branch out and sometimes even outgrow their environment

70 Choosing where to split

• earlier you learned that classification trees used divide-and-conquer to identify splits that create the most pure or homogenous partitions
• let’s see how this works in practise

Decision Tree Split Optiona A

and

Decision Tree Split Optiona B

• to see this in practise, let’s consider our tree being built with lending data on applicants credit and requested loan amounts
• for each of these predictors the algorithm splits feature values and then calculates the purity of the resulting partitions
• the split that produces the most pure partitions will be used first
• split ‘a’ divides the data into partitions with high and low credit scores
• while split ‘b’ divides the data into large and small loan amounts
• though split ‘b’ results in one big homogemous partition, it’s other partition is very mixed
• in comparison split ‘a’ results into partitions that are both relatively pure for this reason tree will used split ‘a’ first
• it then continues to divide-and-conquer always using the split that results in the most homogenous partitions
• as the tree continues to grow, it results in more smaller and homogenous partitions as shown in below image

71 Axis-parallel splits

Decision Tree Axis Parallel 1

• you may have noticed however, that there is an easier way to create a set of perfectly pure partisions
• simply draw a diagonal line between the two outcomes

Decision Tree Axis Parallel 2

• the reason the tree did not discover this itself is that a diagonal split would require the tree to discover combinations of two or more features which is not in the divide-and-conquer process
• instead a decision tree always creates what are called axis-parallel splits
• because the splits line up with the axes
• unfortunately this creates a potential weakness of decision trees, they can be overly complex for some patterns in data

Decision Tree Bigger Tree 2

• generally speaking decision trees have a tendency to become complex very quickly
• a tree can happily divide-and-conquer until it classifies every example correctly
• or until it runs out of feature values to split upon
• when a tree has grown overly large and overly complex, it may experience the problem of overfitting
• rather than modeling the most important trends in data, a tree that has overfitted to the data tends to model the noise
• it is focusing on extremely subtle patterns that may not apply more generally
• more so than any other ML algorithms classification trees have this tendency to overfit the datasets they’re trained on

72 Evaluating model performance

• when a machine learning model has been overfitted to its training dataset you must take care not to overestimate how the model will perform in the future
• just because it perfectly classifies every training example correctly does not mean it will do so on unseen data
• thus it is important to simulate on unseen future data by constructing a test dataset that the algorithm cannot use while growing the tree

Decision Tree Test Set

• a simple method for constructing test sets involves holding out a smaller random portion of the full dataset
• this is a fair estimate of the trees performance
• if the tree performs much more poorly on the test set than the training set it suggests that the model may have been overfitted

73 Why do some branches split?

A classification tree grows using a divide-and-conquer process. Each time the tree grows larger, it splits groups of data into smaller subgroups, creating new branches in the tree.

Given the following groups to divide-and-conquer, which one would the algorithm prioritize to split first?

ANSWER THE QUESTION

Possible Answers The group with the largest number of examples.

The group creating branches that improve the model’s prediction accuracy.

The group it can split to create the greatest improvement in subgroup homogeneity. (answer)

The group that has not been split already.

Correct! Divide-and-conquer always looks to create the split resulting in the greatest improvement to purity.

74 Creating random test datasets

Before building a more sophisticated lending model, it is important to hold out a portion of the loan data to simulate how well it will predict the outcomes of future loan applicants.

As depicted in the following image, you can use 75% of the observations for training and 25% for testing the model.

Decision Tree Test Set

The sample() function can be used to generate a random sample of rows to include in the training set. Simply supply it the total number of observations and the number needed for training.

Use the resulting vector of row IDs to subset the loans into training and testing datasets.

The dataset loans is in your workspace.

Apply the nrow() function to determine how many observations are in the loans dataset, and the number needed for a 75% sample.

Use the sample() function to create an integer vector of row IDs for the 75% sample. The first argument of sample() should be the number of rows in the data set, and the second is the number of rows you need in your training set.

Subset the loans data using the row IDs to create the training dataset. Save this as loans_train.

Subset loans again, but this time select all the rows that are not in sample_rows. Save this as loans_test

# Determine the number of rows for training
nrow(loans)

## [1] 39732

#[1] 39732

nrow(loans)*0.75

## [1] 29799

#[1] 29799

# Create a random sample of row IDs
sample_rows <- sample(nrow(loans), nrow(loans)*0.75)
head(sample_rows)

## [1] 12187 35294 22670 11288 29740 18788

#[1] 32740  8511 37884  3580 34854 39161

length(sample_rows)

## [1] 29799

#[1] 29799

# Create the training dataset
loans_train <- loans[sample_rows,]

# Create the test dataset
loans_test <- loans[- sample_rows,]

Amazing work! Creating a test set is an easy way to check your model’s performance.

75 Building and evaluating a larger tree

Previously, you created a simple decision tree that used the applicant’s credit score and requested loan amount to predict the loan outcome.

Lending Club has additional information about the applicants, such as home ownership status, length of employment, loan purpose, and past bankruptcies, that may be useful for making more accurate predictions.

Using all of the available applicant data, build a more sophisticated lending model using the random training dataset created previously. Then, use this model to make predictions on the testing dataset to estimate the performance of the model on future loan applications.

The rpart package is loaded into the workspace and the loans_train and loans_test datasets have been created.

Use rpart() to build a loan model using the training dataset and all of the available predictors. Again, leave the control argument alone.

Applying the predict() function to the testing dataset, create a vector of predicted outcomes. Don’t forget the type argument.

Create a table() to compare the predicted values to the actual outcome values.

Compute the accuracy of the predictions using the mean() function.

# Grow a tree using all of the available applicant data
loan_model <- rpart(outcome ~ ., 
                    data = loans_train, 
                    method = "class", 
                    control = rpart.control(cp = 0))

# Make predictions on the test dataset
loans_test$pred <- predict(loan_model,loans_test, type = 'class')
"
        
          default repaid
  default    8330    251
  repaid     1277     75

"

## [1] "\n        \n          default repaid\n  default    8330    251\n  repaid     1277     75\n\n"

# Examine the confusion matrix
table(loans_test$outcome, loans_test$pred)

##          
##           default repaid
##   default    8302    207
##   repaid     1356     68

# Compute the accuracy on the test dataset
mean(loans_test$outcome == loans_test$pred)

## [1] 0.8426457

#[1] 0.8461693

Awesome! How did adding more predictors change the model’s performance?

76 Conducting a fair performance evaluation

Holding out test data reduces the amount of data available for growing the decision tree. In spite of this, it is very important to evaluate decision trees on data it has not seen before.

Which of these is NOT true about the evaluation of decision tree performance?

ANSWER THE QUESTION

Possible Answers Decision trees sometimes overfit the training data.

The model’s accuracy is unaffected by the rarity of the outcome. (answer - Not true, rarity of data impacts the model’s performance)

Performance on the training dataset can overestimate performance on future data.

Creating a test dataset simulates the model’s performance on unseen data.

Right! Rare events cause problems for many machine learning approaches.

77 Tending to classification trees

• in the previous video you learned that decision trees have a tendency to grow overly large and complex very quickly
• if this would happen to trees in your yard you would be outside with clippers looking to trim away some of the excess greenery
• grooming healthy classification trees likewise requires this kind of attention
• in this lesson you’ll learn about pruning strategies which help ensure the trees are just right, not too large not too small

78 Pre-pruning

• one method to prevent trees from becoming too large involves stopping the growing process early
• this is known as prepruning
• perhaps the simplest approach to pre-pruning stops divide-and-conquer once the tree reaches a predefined size

• the figure below shows a tree that has been stopped early because it reached the maximum depth of three levels

• see image C:_preprune1.png

• another pre-pruning method requires a minimum number of observations at a node in order for a split to occur

• for example, the figure below stops the tree from growing any branch with fewer than 10 observations

• see image C:_preprune2.png

• both of these pre-pruning strategies prevent the tree from growing too large
• however, a tree stopped too early may fail to discover very subtle or important patterns it might have discovered later
• to address this concern it is also possible to grow a very large tree knowing that it will be overly complex, but then prune it back to reduce the size

• this is known as post pruning

79 Post-pruning

• in post pruning, nodes and branches only with a minor impact on the tree’s overall accuracy are removed after the fat ??
• the below figure, illustrates a tree that grew four levels deep but had a branch pruned away because its presence did not substantially improve the classification accuracy

Decision Tree Post Prune 1

• the relatioinship between the trees complexity and accuracy can be depicted visually as illustrated here

Decision Tree Post Prune 2

• as the tree becomes increasingly complex, the model makes fewer errors
• however, though the performance improves a lot at first it then improves only slightly for the later increases in complexity
• this trend provides insight into the optimal point at which to prune the tree
• simply look for the point at which the curve flattens
• the horizontal dotted line identifies the point at which the error rate become statistically similar to the most complex model
• typically you should prune the tree at the complexity level that results in classification error rate just under this line

80 Pre- and post-pruning with R

• the rpart decision tree package provides a function for creating this visualization as well as performing pre and post pruning
• pre-pruning is performed when building the decision tree model
• the rpart.control function can be supplied with maxdept parameter that controls the maximum depth of the decision tree
• or a minsplit parameter that dictates the minimum number of observations a branch must contain inorder for the tree to be allowed to split
• then simply supply the resulting control object to the rpart function when building the tree

81 pre-pruning with rpart

How to build rpart model and prune the tree.

library(rpart)
prune_control = rpart.control(maxdepth = 30, minsplit = 20)

m = rpart(repaid ~ credit_score + request_amt,
          data = loans,
          method = 'class',
          control = prune_control)

• post pruning has to be applied to a decision tree model that has been previously built
• the plotcp function will generate a visulization of error rate vs model complexity
• which provides insight to the optimal cut point for pruning
• when this value has been identified, it can be supplied to the prune functions complexity parameter cp to create a simpler pruned tree

# post-pruning with rpart
m = rpart(repaid ~ credit_score + request_amt,
          data = loans,
          method = 'class')

plotcp(m)

m_pruned = prune(m, cp = 0.20)

82 Preventing overgrown trees

The tree grown on the full set of applicant data grew to be extremely large and extremely complex, with hundreds of splits and leaf nodes containing only a handful of applicants. This tree would be almost impossible for a loan officer to interpret.

Using the pre-pruning methods for early stopping, you can prevent a tree from growing too large and complex. See how the rpart control options for maximum tree depth and minimum split count impact the resulting tree.

The rpart package is loaded into the workspace.

Add a maxdepth parameter to the rpart.control() object to set the maximum tree depth to six. Leave the parameter cp = 0. Pass the result of rpart.control() as the control parameter in your rpart() call.

See how the test set accuracy of the simpler model compares to the original accuracy of 58.3%. - First create a vector of predictions using the predict() function. - Compare the predictions to the actual outcomes and use mean() to calculate the accuracy.

Add a minsplit parameter to the rpart.control() object to require 500 observations to split. Again, leave cp = 0.

Again compare the accuracy of the simpler tree to the original.

# Grow a tree with maxdepth of 6
loan_model <- rpart(outcome ~ ., data = loans_train, method = "class", 
                    control = rpart.control(cp = 0, maxdepth = 6))

# Compute the accuracy of the simpler tree
loans_test$pred <- predict(loan_model, loans_test, type = "class")
mean(loans_test$pred == loans_test$outcome)

## [1] 0.8559348

# Grow a tree with minsplit of 500
loan_model2 <- rpart(outcome ~ ., data = loans_train, method = "class", control = rpart.control(cp = 0, minsplit = 500))

# Compute the accuracy of the simpler tree
loans_test$pred2 <- predict(loan_model2, loans_test, type = "class")
mean(loans_test$pred2 == loans_test$outcome)

## [1] 0.8566395

Nice work! It may seem surprising, but creating a simpler decision tree may actually result in greater performance on the test dataset.

83 Creating a nicely pruned tree

Stopping a tree from growing all the way can lead it to ignore some aspects of the data or miss important trends it may have discovered later.

By using post-pruning, you can intentionally grow a large and complex then prune it to be smaller and more efficient later on.

In this exercise, you will have the opportunity to construct a visualization of the tree’s performance versus complexity, and use this information to prune the tree to an appropriate level.

The rpart package is loaded into the workspace, along with loans_test and loans_train.

Use all of the applicant variables and no pre-pruning to create an overly complex tree.

Make sure to set cp = 0 in rpart.control() to prevent pre-pruning.

Create a complexity plot by using plotcp() on the model.

Based on the complexity plot, prune the tree to a complexity of 0.0014 using the prune() function with the tree and the complexity parameter.

Compare the accuracy of the pruned tree to the original accuracy of 58.3%. To calculate the accuracy use the predict() and mean() functions.

# Grow an overly complex tree
loan_model <- rpart(outcome ~ ., data = loans_train, method = "class", control = rpart.control(cp = 0))

# Examine the complexity plot
plotcp(loan_model)

# Prune the tree
loan_model_pruned <- prune(loan_model, cp = 0.0014)

# Compute the accuracy of the pruned tree
loans_test$pred <- predict(loan_model_pruned, loans_test, type = "class")
mean(loans_test$outcome == loans_test$pred)

## [1] 0.8566395

Great job! As with pre-pruning, creating a simpler tree actually improved the performance of the tree on the test dataset.

84 Why do trees benefit from pruning?

Classification trees can grow indefinitely, until they are told to stop or run out of data to divide-and-conquer.

Just like trees in nature, classification trees that grow overly large can require pruning to reduce the excess growth. However, this generally results in a tree that classifies fewer training examples correctly.

Why, then, are pre-pruning and post-pruning almost always used?

ANSWER THE QUESTION

Possible Answers Simpler trees are easier to interpret

Simpler trees using early stopping are faster to train

Simpler trees may perform better on the testing data

All of the above (answer)

Yes! There are many benefits to creating carefully pruned decision trees!

Seeing the forest from the trees

• consider the ways decision trees parallel to trees in natural environment
• from a root node that grows into branches and leaf nodes that sometimes need pruning you might think that we might have exhausted the tree metaphors
• infact there is one more
• just as living trees can be grouped as a forest a number of classification trees can be combined into a collection known as decision tree forest
• for the reasons that you will soon see these forests are among the most powerful machine learning classifiers yet remain remarkable efficient and easy to use

85 Understanding random forest

Decision Tree Forest

• because of their combination of versatility and power decision tree forests have become one the most popular approaches for classification
• this power does not come from a single tree that has grown large and complex but rather from a collection of smaller simpler trees that together reflect the data’s complexity
• each of the forests tree is diverse and may reflect some suble pattern in the outcome to be modeled
• generating this diversity is the key to building powerful decision tree forests
• however, if you were to grow one hundred trees on the same set of data you’d have one hundred times the same tree
• growing diverse trees requires the growing conditions to be varied from tree to tree
• this is done by allocating each tree a random subset of data
• one may receive a vastly different training set than another
• the term random forest refers to a specific growing algorithm in which both the features and examples may differ from tree to tree

86 Making decisions as an ensemble

Decision Tree Ensemble 1

• it seems somewhat counter intuitive to that a group of trees built on small random subsets of the data could perform any better than a single really complex tree that had the benefit of learning the entire dataset
• but the force power is based on the same principles that govern successful teamwork in business or on the athletic field
• in this cases it is certainly advantageous to have team members that are extremely good at some tasks
• however, these people typically have weaknesses in other areas
• for this reason it is even better for the team to have members with complimentary skills even if non of the members is especially strong good team work usually wins

Decision Tree Ensemble 2

• ML methods like random forests that apply this principle are called ensemble {aun somble} methods
• all ensemble methods are based on the principle that weaker learners become stronger with team work
• in our random forest each tree is asked to make a prediction and the groups overall prediction is determined by a majority vote
• though each tree major represent a narrow portion of the data the overall consensus is strengthened by these diverse perspectives

87 Random forests in R

• the R package randomForest implements the random forest algorith
• the function offers two parameters of node
• the number of trees to include in the forest
• setting this sufficiently large will ensure good representation of the complete set of data
• don’t worry even with a large number of trees, the model runs relatively quickly as each tree uses only a small portion of the full dataset
• mtry is the number of features selected at random for each tree
• by default it uses the square root of the total number of predictors
• generally, it is okay to leave this parameter as-is
• as usual the predict function uses the model to make predictions

88 building a simple random forest

library(randomForest)

m = randomForest(repaid ~ credit_score + request_amt, data = test_loans,
ntree = 500, # number of trees in the forest
mtry = sqrt(p)) # number of predictors (p) per tree)


# making predictions from a random forest
p = predict(m, test_data)

89 Understanding random forests

Groups of classification trees can be combined into an ensemble that generates a single prediction by allowing the trees to ‘vote’ on the outcome.

Why might someone think that this could result in more accurate predictions than a single tree?

ANSWER THE QUESTION

Possible Answers Each of tree in the forest is larger and more complex than a typical single tree.

Every tree in a random forest uses the complete set of predictors.

The diversity among the trees may lead it to discover more subtle patterns. (answer)

The random forest is not affected by noisy data.

Yes! The teamwork-based approach of the random forest may help it find important trends a single tree may miss.

90 Building a random forest model

In spite of the fact that a forest can contain hundreds of trees, growing a decision tree forest is perhaps even easier than creating a single highly-tuned tree.

Using the randomForest package, build a random forest and see how it compares to the single trees you built previously.

Keep in mind that due to the random nature of the forest, the results may vary slightly each time you create the forest.

# Load the randomForest package
library(randomForest)

# Build a random forest model
loan_model <- randomForest(outcome ~ ., data = loans_train)

The above gives below error:

Error in randomForest.default(m, y, …) : NA/NaN/Inf in foreign function call (arg 1)

The Chr predictors need to be converted to factors to remove the above error. Let’s try with two predictors loan_amount and credit_score

# Convert chr values to factor for two predictors
library(randomForest)

## Warning: package 'randomForest' was built under R version 3.5.2

## randomForest 4.6-14

## Type rfNews() to see new features/changes/bug fixes.

## 
## Attaching package: 'randomForest'

## The following object is masked from 'package:ggplot2':
## 
##     margin

## The following object is masked from 'package:dplyr':
## 
##     combine

table(loans_train$loan_amount)

## 
##   HIGH    LOW MEDIUM 
##   8361   7378  14060

str(loans_train$loan_amount)

##  chr [1:29799] "LOW" "LOW" "LOW" "LOW" "HIGH" "MEDIUM" "HIGH" "MEDIUM" ...

loans_train$loan_amount = as.factor(loans_train$loan_amount)
table(loans_train$loan_amount)

## 
##   HIGH    LOW MEDIUM 
##   8361   7378  14060

str(loans_train$loan_amount)

##  Factor w/ 3 levels "HIGH","LOW","MEDIUM": 2 2 2 2 1 3 1 3 1 2 ...

table(loans_train$credit_score)

## 
## AVERAGE    HIGH     LOW 
##   20283    5936    3580

str(loans_train$credit_score)

##  chr [1:29799] "AVERAGE" "AVERAGE" "AVERAGE" "LOW" "HIGH" "AVERAGE" ...

loans_train$credit_score = as.factor(loans_train$credit_score)
table(loans_train$credit_score)

## 
## AVERAGE    HIGH     LOW 
##   20283    5936    3580

str(loans_train$credit_score)

##  Factor w/ 3 levels "AVERAGE","HIGH",..: 1 1 1 3 2 1 1 3 1 2 ...

# Convert outcome to numberic while passing to randomForest 
loan_model <- randomForest(outcome ~ loan_amount + credit_score, 
                           data = loans_train,
                           ntree = 500)


loan_model

## 
## Call:
##  randomForest(formula = outcome ~ loan_amount + credit_score,      data = loans_train, ntree = 500) 
##                Type of random forest: classification
##                      Number of trees: 500
## No. of variables tried at each split: 1
## 
##         OOB estimate of  error rate: 14.2%
## Confusion matrix:
##         default repaid class.error
## default   25569      0           0
## repaid     4230      0           1

Make the prediction for test data

# First convert the predictors to factors
test_predictors = loans_test %>% select(loan_amount, credit_score)
test_predictors$loan_amount = as.factor(test_predictors$loan_amount)
test_predictors$credit_score = as.factor(test_predictors$credit_score)

# Predict using random forest model
loans_test$pred <- predict(loan_model, test_predictors)

# Compute the accuracy of the random forest
mean(loans_test$outcome == loans_test$pred)

## [1] 0.8566395

Repeat by adding more predictors. To Do: write a function to convert chr variables to factors if they have unique levels/categories

Wow! Great job! Now you’re really a classification pro! Classification is only one of the problems you’ll have to tackle as a data scientist. Check out some other machine learning courses to learn more about supervised and unsupervised learning.